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This study identified the skills and tasks practiced by instructional design personnel within Army military 
training organizations. The purpose of the study was to discover trending skills and tasks utilized by 
instructional systems specialists employed at the United States Army Aviation Center of Excellence 

(USAACE), Fort Rucker. Trends were based on Instructional Designer (ID) competencies established by 

the International Board of Standards for Teaching, Performance, and Instruction (IBSTPI) but allowed for 

additional trends to emerge. A descriptive case study research design was used to collect and analyze the 
data within this study. Seventeen (17) instructional systems specialists (ISSs) were interviewed. Member 
checking and ISS expert review was used to validate the trends found within the data. Of the twenty-two 
(22) established Instructional Design IBSTPI Competencies thirteen (13) were identified as practiced by 
ISSs within the USAACE training organization during the time of the interview. Of those thirteen 
competencies three (3) were identified as only partially practiced by ISSs. Two (2) additionally 
competencies not mentioned with the IBSTPI list of competencies emerged: Mentoring Others and 
Teaching or Classroom Management. Additionally, one major unexpected trend regarding the perception 
of ID performance emerged from the data. The amount and degree of educational background in 
comparison to previous military service experience indicated different approaches toward design 
project/work assignments. 
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Chapter One: Introduction 

Professional Instructional Designers (IDers) recognize that most individuals cannot 
answer the following questions: What is an IDer? What does an [Der do? Even aforementioned 
designers have a difficult time explaining the depth and breadth of this career field so others may 
understand. “Unlike many other fields, [instructional design] is constantly changing due to 
emerging theory, practice, and technological innovations” (Kelly, 2016, p. 5). As such, the 
potential of [Der is vastly under-utilized. For this reason, many within the field have attempted to 
explain the practices of the profession through study and reporting. 

For organizations to maximize the value and organizational success of persons filling 
design positions, it is important for organizations to understand the abilities and capabilities of 
the IDer. Individuals with the education and experience of Instructional Design (ID) have the 
skills to identify areas of inefficiencies in an organization, as well as to identify various solutions 
for resolving them. IDers are analytical experts who determine the best solutions to any problem, 
whether it is doctrinal, organizational, material, leadership or management development, 
personnel development, facility and infrastructure, and/or training. If the solution is relevant to 
training needs, then the IDer can determine the most effective training strategy for the audience 
intended to receive it. However, the role of an IDer, particularly as a change agent, continues to 
be misunderstood or overlooked (Rowland, 2005). If organizational leaders do not understand 
what IDers can contribute, they lose a powerful avenue toward success. 

Historically, many studies have been conducted with the intent of providing insight into 
the perceptions and practices of IDers. The available research focuses mainly on two overarching 
topics. The first topic is the difference in design performance between novice and expert 


designers (Hannum, 2005; Osguthorpe & Osguthorpe, 2007; Rowland, 1992; Tracy & Boling, 


2014). The second topic answers the question of how IDers perform their jobs, in comparison to 
what they were taught to perform within their respective academic programs (Cox & 
Osguthorpe, 2003; Denner & Spector, 2007; Schwier et al., 2004; Tracey & Hutchinson, 2013). 
The conclusion of those studies lead to a larger question: should academic programs be re- 
designed to prepare [Ders more authentically as to how ID is reportedly practiced in the field, as 
well as to support the development of more expert level skills within novices? These studies, 
however, are conducted primarily by academic/collegiate researchers. Furthermore, the research 
is primarily representative of ID practices occurring in academic or corporate organizations, 
limiting the data to those fields. One large employer of IDers is consistently disregarded 
throughout the existing literature. The Department of Defense (DOD) is the largest global 
employer worldwide (WorldAtlas, 2020). Many positions within the DOD are civilian 
instructional designers. However, the perceptions and practices of [Ders in this sector remain 
unknown. 

Overall, researchers of the perceptions and practices of ID are in agreement. They concur 
that novice designers spend much more time conducting analyses and gathering data, before 
engaging in design efforts. Expert designers, on the other hand, engage in design tasks quickly in 
the design process, by conducting analysis and design conjointly. Researchers also agree that 
while academic programs teach instructional design as a linear five step (minimum) process with 
specific performance tasks embedded in each phase, ID rarely, if ever, occurs linearly. It often 
skips or minimizes the amount of time in certain tasks (Gibbons, 2003; Kenny et al., 2005; 
Tracey & Boling, 2014; Tracey & Hutchinson, 2013). 

Based on these conclusions, it is unclear whether or not including the military 


instructional designer’s, referred to in this study as Instructional Systems Specialists (ISS), 


demographic would alter current data in the field. If so, it is possible that understanding 
government and military ID practices could impact current conversations about designing and 
implementing academic ID programs. Additionally, this could lead to new perceptions of ID 
culture, specifically within a uniquely structured organization that differs greatly from academic 
and non-government organizations. Much information could be added to the research and 
understanding of this field if these gaps were addressed. 
Problem Statement 

Currently, government and military training organizations do not seem to understand the 
roles of IDers within their ranks. This is evident by the fact that, more often than not, IDers or 
instructional systems specialists, instructors, and training specialists are used interchangeably 
(Department of the Navy United States Marine Corps, 2009). In many cases the designer is 
utilized as data entry personnel or as contract managers instead of as designers. Personnel who 
are not in an ID or ISS position and do not fulfill the requisite background in education are 
required to design and develop training courseware, which include the following components: 
PowerPoint presentations, lesson plans, student handouts, and assessments. These same 
government institutions often hire individuals with no educational background specific to ID into 
ISS positions (Department of the Navy United States Marine Corps, 2009). Moreover, there is no 
date of expiration on educational requirements being met. In short, a person who has completed 
only the required 24 academic credit hours in a physical education degree in 1980 is technically 
as eligible for hire as a person who has acquired a doctorate in ID in 2020. 

Unfortunately, there is little to no professional development or performance support when 
the ISS is asked to make course modifications in response to unit and Army mission needs. This 


is due to the lack of understanding of the ID profession (Klein & Kelly, 2018) by leaders and 


practitioners within government and military training organizations. This results in training 
decisions being made with little consideration to design option solutions that can improve the 
quality of training and maximize utilization of course resource constraints. 

This projected study addressed the lack of organizational knowledge of ID perceptions 
and practices of instructional systems specialists (ISS). The study answered the question of what 
an ISS does and how they perform their jobs. This study also elucidated the specific educational 
achievement within the field of ID in support of professional position performance capabilities. 
This study did not depart from the information provided by the current literature. The addition of 
this study attempted to fill an informational gap in the literature on ID practice. This study 
provided further insight into the content and design of collegiate ID programs as those 
conversations continue among academics and professionals. 

Purpose of the Study 

This research directly responded to a review of the topic literature. It ruminated on 
questions concerning the academic and corporate or non-government organizational perceptions 
and practices of instructional designers. As I found very little data existing in the literature on the 
practice of ID regarding perceptions and practices of IDers in military training organizations, this 
study served to introduce that baseline data for consideration and future research. For this study, 
baseline data, was considered the identification of education degrees held by Instructional 
Systems Specialists (ISSs), tasks routinely performed, and perceptions of expertise required for 
tasks performed. 

Research Questions 
The research questions addressed within this study served to eliminate an unintentional 


bias toward that of [Ders in military training environments. By introducing the foundational 


perceptions and practices of ISS, this study broadened the view of future professionals, 
academics, and researchers. The research questions for this study were as follows: 

RQI1: What type of education and/or training have Instructional Systems Specialists 
(ISSs) practitioners received as Instructional Designers (IDers)? 

RQ?2: What International Board of Standards for Training, Performance, and Instruction 
(IBSTPI) Instructional Design (ID) tasks are performed most regularly by Instructional Systems 
Specialists (ISSs) practitioners? 

RQ3: What is the perceived level of expertise requisite for performance of Instructional 
Design (ID) tasks by Instructional Systems Specialists (ISSs)? 

Definition of Terms 

In an effort to clarify key terms used throughout this research, I have created a list of 
frequently used terms by professional and military organizations, which were referred to 
frequently in this study. 

Instructional Design (ID) 

ID is the creation of instructional materials. This field goes beyond simply creating 
teaching materials, it requires that the Instructional Designers know how students learn and what 
materials and methods will most effectively help individuals achieve their academic goals. The 
principles of ID consider how educational tools should be designed, created and delivered to any 
learning group, from grade school students to adult employees across all industry sectors. 
Instructional Systems Specialists (ISS) 

The ISS (1750 series) is the IDer position as titled within the Army organization. This 


position requires a minimum of 24 academic credit hours in education. 


Training Specialist 

The Training Specialist (1712 Series) is the instructor or instructor/writer position within 
the Army organization. This position has no educational requirements, but often acts as content 
Subject Matter Experts (SMEs) because of previous military experience. 

Training Developer 

The Training Developer is the umbrella term used within the Army training organization 
which encompasses both the ISS and the training specialist. 
Training and Doctrine Command (TRADOC) 

Training and Doctrine Command (TRADOC) is located at Fort Eustis in Virginia. This is 
the command organization that oversees and provides regulatory guidance to all Army training 
installations. Its mission is to recruit, train, and educate the Army, driving constant improvement 
and change to ensure the Total Army can deter, fight, and win on any battlefield now and into the 
future (Townsend, 2018). 

Career Program 32 

The TRADOC program that provides funding for career development opportunities to all 
civilian personnel that hold an educational or training position. Each installation has its own 
Career Program 32 manager. 

Army University (ArmyU) 

ArmyU is located at Fort Leavenworth in Kansas. The ArmyU aligns most of the 
Army’s military education programs under a unified academic structure. It reinvests and 
transforms the Army’s education programs to improve individual performance and increase 


readiness, provides insight on on-going initiatives related to Army education, and resources 


challenges associated with improving the quality and rigor of Army educational programs (The 
Army University, 2017, para.1). 
Significance of the Study 

This study provided knowledge specific to a professional field about its practices within 
another work structure that, to date, has been overlooked by existing research. As previously 
mentioned, research reflecting the performance of ID tasks does not exist. Currently, research 
only reflects job performance in academic or industrial settings. The ID literature suggests that 
knowing how IDers perform in academia and industry is important because the data influences 
learning theory (Kenny et al., 2005); maintains a current and accurate view of the profession as it 
evolves (Byun, 2001; Gibbons, 2013; Patel, 2010; Reiser, 2001; Tracey & Boling, 2014); 
maintains updated job and task analyses that lead to changes in educational programs (Cox & 
Osguthorpe, 2003); and prepares Instructional Designers for different career environments 
(Dennen & Spector, 2007; Larson & Lockee, 2009). The same suggestions holds true for 
knowing what Instructional Designers do in military training environments, which exist in 
unique settings with distinctive training needs. 

The literature on ID practice and perception does not routinely note that the ID profession 
evolved from a need for training in the armed forces during WWII and that the Army in 
particular was a significant influential factor to the establishment of ID as a profession. The ID 
practice literature that currently exists, however, does influence decisions about the profession in 
terms of the design of academic programs, human capital strategy planning, learning and design 
theory creation, and identification of solutions for future issues in the profession (Cox & 
Osguthorpe, 2003), as well as professional competency development (IBSTPI, 2020). Missing 


data specific to government or military entities means that decisions regarding the profession are 


not completely informed. Inclusion of the perceptions and practices of military training 
Instructional Designers influences the evolution and improves decisions that impact the 
profession as a whole. 
Conclusion 

This study encouraged wider educational and performance considerations when training 
new instructional designers for the workforce. The knowledge potentially gained provided Army 
leaders and managers insight into Army hiring actions within the ID field. Additionally, this 
study provided insight to potential Army personnel management advantages that may be by- 
passed from lack of understanding, particularly in a military training environment that is re- 
designing the methods and media it uses to educate its soldiers due to budgetary constraints, 
global events, and changes in mission operational locations (i.e., large scale combat or multi- 


domain operations). 


Chapter Two: Literature Review 

The main focus of this thesis is the practice and perception of Instructional Designers 
(IDers) working in military training environments. This chapter begins with an introduction to 
the field of instructional design; a history and use of the Analysis, Design, Development 
Implementation, and Evaluation (ADDIE) process; an explanation of the International Board of 
Standards, Training, and Performance and Instruction (IBSTPI) competencies for instructional 
designers (IDers); and a look at a composite of position descriptions for Army Instructional 
Systems Specialists (ISSs). The chapter concludes with a summary of the two primary topics 
derived from the reviewed literature of Instructional Design (ID) practices and perceptions. 

Instructional Design (ID) 

Instructional design (ID) has evolved to stay relevant and current with technology for 
educational purposes (Byun, 2001; Patel, 2010; Reiser, 2001; Tracey & Boling, 2014). Because 
the profession has continued to advance, however, it has become increasingly difficult to provide 
an adequate and comprehensive definition. The profession of ID goes by many names such as 
Instructional Systems Design (ISD); Instructional Systems Technology (IST); Systems Approach 
to Training (SAT); and most recently, Educational Technology (Klein & Kelly, 2018; Larson & 
Lockee, 2004; Reiser, 2001). Many researchers have tried to define instructional design over the 
years as the art and science of constructing learning environments through the considerations of 
technology and learning theory in order to improve student knowledge, skills, and abilities from 
its original state (Merrell et al., 1996; Siemens, 2002; Smith & Ragan, 2005; Reiser & Dempsey, 
2007). For the purposes of this study, the term instructional design is used and encompasses all 


variations of the titled profession. 
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The Army’s Training and Doctrine Command (TRADOC) regulation, Army Learning 
Policy and Systems, TR-350-70, does not offer a definition for ID. The purpose of this 
regulation, “is to support the Army by regulating production, implementation, and evaluation 
practices for effective learning management and to specify required enabling systems” (U.S. 
Department of the Army, 2017b, p. 11) for all United States Army training organizations. The 
regulation does, however, define Army Learning as, “the act of acquiring, maintaining, or 
improving knowledge, skills, and attitudes to achieve required performance. It is the combination 
of training, education, and experience” (U.S. Department of the Army, 2017b, p. 21). Training is 
defined as psychomotor learning, while education is considered cognitive learning, and 
experience is the practical application of both. In the attempt to provide sound ID guidance, the 
TRADOC Regulation 350-70 lists descriptions of products and events within the instructional 
design process (U.S. Department of the Army, 2017b, p. 31). Products include: Critical Task Site 
Selection Boards (CTSSBs), Individual Critical Task Lists (CTLs), Learning Objectives, Career 
Maps, and Lesson Plans. The guidance also provides a detailed description of the Analysis, 
Design, Development, Implementation, and Evaluation (ADDIE) Process. This process is 
discussed in more detail in a later section of this chapter. 

Currently, there is a subtle, discernable shift from a technological, or audiovisual, focus 
toward a focus on the systematic and artful science of environment and material design (Kelly, 
2016; Reiser, 2001). However, even these expert definitions of the field may be lacking. The 
definitions seem unable to depict the intricacies of applicable theoretical considerations and 
practical decisions made throughout the design process as they relate to the content, audience, 


and desired outcome. 
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Tatar (2007) believes that design is not about problem solving; rather, it is about goal 
balancing. Tatar explains that design tensions do not set boundaries or simplify the problem, but 
provide a framework for creating a space of relevance. Rather than focusing on simplifying a 
problem, design tensions provide a framework in which the designer can manage complexity and 
trade-offs (Tatar, 2007). Designers are change agents rather than simply creators of artifacts and 
experiences. Personally, I often describe ID to those not familiar with the field as the attempt to 
match technology, instructional methods, and environment in order to package the right message, 
for the right content, and for the right audience, so they can retain and recall information at the 
correct time better, faster, stronger, and longer. 

Instructional Design (ID) History 

While having a general understanding of ID is important, for this study it was imperative 
to know where the field originated. This study focused on ID task performance within Army 
training organizations. However, based on the literature available on the subject, most 
practitioners in the ID profession seemed to be unaware that work was available to them in the 
government/military sector. 

Because the history of ID is unknown, most ID professionals assume that the profession 
is academic, industry-originated, and industry-oriented. Additionally, many scholars who 
research this topic fail to understand the importance of including military organizations within 
their sample demographic. The ID profession was originated for the needs of the military, yet 
there is a severe lack of literature surrounding those in the ID profession within the environment 
for which the role was created. 

During World War IJ, the United States military was confronted with the need to prepare 


soldiers for deployment and train them in complex weapons and maneuvers more quickly than in 


LZ 


previous decades (Brooker, 1946). In response to that need, the military hired expert researchers 
in the fields of psychology and education. Between 1941 and 1945, these researchers were tasked 
by the Division of Visual Aid for War Training in the United States Department of Education 
with designing instruction, using the latest audio-visual technologies, to train larger numbers of 
soldiers quickly (Brooker, 1946; Reiser, 2001; Walcutt & Schatz, 2019). Following the 
establishment of the Division of Visual Aid for War Training and the conclusion of the war, 
researchers began to study theories of learning. 

The timeline below (see Table 1) presents the advent of ID in the 1940s and the 
development of a multitude of subsequent learning theories until it became an independent 
profession of its own. 


Table 1 


Timeline to Instructional Design (ID) 


1941 1956 1962 1963 1965 1968 1975 
Division of Skinner’s Mager’s Glaser’s Gagne’s Atkinson’s Academia 
Visual Aid | “The Science Preparing “Criterion Conditions | and Shiffrin’s formally 

for War of learning Objectives Referenced | of Learning | Information introduces 
Training and the Art of for Measures” Processing ID 
Established Teaching” Programmed termed Model programs 
Published Instruction coined 
& 
Bloom’s 
Taxonomy of 
Educational 
Objectives 
created 


Note: Table was influence by research from Brooker (1946), Larson & Lockee (2004), Reiser 
(2001), Schunk (2012). 


This timeline highlights the birth of ID within the military, which led to prominent roles 
today, as well as the development of new theories and research within the field. Reiser (2001) 


aptly described the historical progression of the field: 
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Immediately after World War II, many of the psychologists responsible for the success of 
the military training programs continued to work on solving instructional problems. 
Organizations such as the American Institutes for Research were established for this 
purpose. During the late 1940s and throughout the 1950s, psychologists working for such 
organizations started viewing training as a system, and developed a number of innovative 
analysis, design, and evaluation procedures. (p. 23) 
The profession, and the research, continued to steadily grow over the decades, making ID an 
integral part of growth, design, and the education process. 
The Army’s ADDIE Process 

While the practice of ID was initiated in 1941 to train soldiers for war, the ADDIE 
process didn’t become a formal practice until 1975. The process was developed by the Centre for 
Education Technology at Florida State University for use with the U.S. Army (Kurt, 2017). This 
process was created by educational researchers and scholars, such as Robert Gagne (1975), in 
order to continue supporting Army training needs (Reiser, 2001; Schunk, 2012). Throughout the 
field of ID, there are a myriad of popular and well-respected design models. At the core of each 
is the ADDIE process (Klein & Kelly, 2018; Kurt, 2017). 

The ADDIE process is defined as “‘a framework used to organize and manage all course 
and curriculum development activities using a disciplined process that ensures classroom 
instruction accomplishes the institutions educational purpose” (U.S. Department of the Army, 
2017b). The ADDIE process attempts to provide focus to an otherwise chaotic and complicated 
effort. It provides structure and program management guidance for the development and 
maintenance of instructional materials (U.S. Department of the Army, 2017b). Since its 


inception, the five phases of the ADDIE process have remained stable, acting as the foundation 
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for other design processes that attempt to offer greater descriptions of how to perform ID. 
Despite the development of other design processes, the Army has always utilized the ADDIE 
process as its design process of choice. For a period of time, Army training organizations used 
the term Systems Approach to Training (SAT) instead of the ADDIE process. The five phases 
remained the same, however, no matter what the process was called. In 2015, with the 
establishment of Army University, the process was once again called ADDIE. Furthermore, the 
process was originally instructed as a step-by-step, linear process but gradually became a cyclical 
and continuous process where practitioners can begin at any phase based on situation and need. 


Figure 1 
The ADDIE Process with Associated Key Products 
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The five phases of the ADDIE process are Analysis, Design, Development, 
Implementation, and Evaluation. The intent of each phase is to inform what actions, 


considerations, procedures are conducted in order to produce training products or problem 
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solutions that effectively address desired training outcomes for specific audiences. These stages 
are presented in Figure 1, which illustrates the ADDIE process with associated key products, the 
stages, and tasks within said stages. 
Analysis Phase 

In the analysis phase the IDer works to collect as much information as they can, 
regarding the training to be developed. “It involves the detailed breakdown and examination of 
jobs, functions, tasks, objectives, and performance measures. It serves as the foundation for all 
learning products and drives the design and development of curricula and learning products” 
(U.S. Department of the Army, 2017b, p. 47). The first analysis—the needs or gap analysis— 
serves to determine whether or not training is the solution to the problem. If training is the 
solution, then the IDer begins collecting and reviewing data from as many sources as possible to 
answer questions about the target audience, the tasks or topics to be trained, and the environment 
in which the training will occur. Products of the analysis phase include: target audience analysis, 
need analysis, and job task analysis. The TRADOC regulation 350-70 includes mission analysis 
and doctrine analysis and also guides IDers during this phase. 
Design Phase 

The design phase is the ability to organize the collected data into neat organized 
compilations of corresponding information. Next, those compilations of information can be 
selected and sequenced in an order that best suits the outcome, audience, and environment. It 
“translates data into an outline for learning, creating a blueprint for learning product 
development, and determines the sequence and how to train” (U.S. Department of the Army, 
2017b, p. 49). The decisions are made as to which levels of learning are appropriate, the number 


and type of activities to be included, the methods and media that best accomplishes the outcome. 
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Products of the design process include: storyboards, learning objectives, test items, methods and 
media selection, environment selection, and content sequence. These products are then used in 
the next phase of the process: development. 
Development Phase 

The development phase consists of producing the required materials for instruction, based 
on the design decisions. The development phase incorporates ideas and decisions made within 
the design phase (U.S. Department of the Army, 2017b). This phase stipulates that validation of 
developed products should occur prior to implementation. The purpose of the validation process 
is to ensure the highest quality of the training products. With strict regulations, the intention of 
validation is also to ensure the appropriate identification of resources in order to avoid 
unnecessary expenses should changes be required following the validation period or as a result of 
triggering events that warrant the changes within a course or lesson (U.S. Department of the 
Army, 2017b). Products of the development phase include: lesson plans, student handouts, 
presentations, training aids, assessments, media selected for the training, and training 
instructional personnel. Once these products are validated, they are then used to put the plan into 
action. 
Implementation Phase 

The implementation phase consists of the delivery of fully developed and validated 
curriculum to classes matching the target audience in designated training environments. This 
particular phase “includes the execution of lesson plans according to its design” (U.S. 
Department of the Army, 2017b, p. 51). Products of the implementation phase include: 
instruction, maintenance of facilities, materials, and equipment, maintenance of current and 


relevant reference materials, and reporting student assessment. 
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Evaluation Phase 
The evaluation phase is ongoing. This phase occurs throughout every phase and acts as 

“quality control for learning and learning product development as well as ensuring the learning 
has achieved intended outcomes. Evaluation is a systematic method to appraise the quality, 
effectiveness, and efficiency of a program, process, product, or procedure” (U.S. Department of 
the Army, 2017b, p. 52). Products of the evaluation phase include: formative and summative 
evaluations. These products are used to monitor and adjust the previous phases as needed to fit 
the needs and designers of the audience, as well as maintaining relevancy in an ever-changing 
environment. 

International Board of Standards, Training, and Performance and Instruction (IBSTPI) 

Instructional Design (ID) Competencies 
In a field that is constantly growing with the changes of technology, standards and 

competencies have been solidified to maintain integrity within the field. The IBSTPI is a non- 
profit organization that works to identify and define those standards and competencies performed 
by instructional professionals throughout industry, academia, and government agencies (IBSTPI, 
2020). IBSTPI develops, validates, publishes, and disseminates the competencies and 
performance statements for several areas, including instructors, training managers, [Ders, and 
evaluators. Competencies include knowledge, skills, and abilities for successfully performing the 
roles required by the position (Koszalka et al., 2013; Richey et al., 2001). The IBSTPI 
competencies for IDers were first established in 1986 but are regularly reviewed and updated to 
remain current and relevant to the field (Byun 2000; Kelly, 2016; Koszalka et al., 2013; Richey 
et al., 2001). The most recent competencies released by IBSTPI include 22 instructional designer 


competencies. Each competency is labeled as either essential, advanced, or managerial. These 
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labels are indicative of the job role the IDer holds. Knowledge, skills, and abilities that should be 
mastered by all IDers are considered essential. Knowledge, skills, and abilities that should be 
mastered by experienced IDers are considered advanced. Knowledge, skills, and abilities 
performed by IDers in managerial or supervisory positions are labeled as managerial (Koszalka 
et al, 2013). Army University, a subsidiary of the U.S. Army Combined Arms Center at Fort 
Leavenworth, Kansas an internally developed list of 21‘ Century Soldier Competencies for the 


IBSTPI competencies in 2015 for the use of instructors and ISSs. IBSTPI reports the following 


(see Table 2) as instructional design competencies. 
Table 2 


IBSTPI Instructional Design Competencies 


Level of 

PROFESSIONAL FOUNDATIONS Expertise 
1. Communicate effectively in visual, oral, and written form. Essential 
2. Apply research and theory to the discipline of instructional design. Advanced 
3. Update and improve knowledge, skills, and attitudes pertaining to the Essential 
instructional design process and related fields. 
4. Apply data collection and analysis skills in instructional design projects. | Advanced 
5. Identify and respond to ethical, legal, and political implications of Essential 
design in the workplace. 

PLANNING AND ANALYSIS 
6. Conduct a needs assessment in order to recommend appropriate design Advanced 
solutions and strategies. 
7. Identify and describe target population and environmental Essential 
characteristics. 
8. Select and use analysis techniques for determining instructional content. Essential 
9. Analyze the characteristics of existing and emerging technologies and Essential 
their potential use. 
DESIGN AND DEVELOPMENT 

10. Use an instructional design and development process appropriate for a Essential 
given project. 
11. Organize instructional programs and/or products to be designed, Essential 
developed, and evaluated. 
12. Design instructional interventions. Essential 
13. Plan non-instructional interventions. Advanced 
14. Select or modify existing instructional materials. Essential 
15. Develop instructional materials. Essential 
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16. Design learning assessment. Advanced 
EVALUATION and IMPLEMENTATION 

17. Evaluate instructional and non-instructional interventions. Advanced 

18. Revise instructional and non-instructional solutions based on data. Essential 


19. Implement, disseminate, and diffuse instructional and non-instructional | Advanced 
interventions. 


MANAGEMENT 
20. Apply business skills to managing the instructional design function. Managerial 
21. Manage partnerships and collaborative relationships. Managerial 
22. Plan and manage instructional design projects. Advanced 


Note: Adapted from Koszalka et al. (2013), and IBSTPI (2020). 

IBSTPI is a highly reputable organization within the field of ID (Byun, 2000). “The 
importance of IBSTPI ID competencies is that they cover the full range of instructional design 
and development positions, including areas defined by the widely-used ADDIE process” (Byun, 
2000, p. 21). The organization and the competencies provide an understanding and set guidelines 
within a confusing field. 

There is no analytical data provided by Army University as to why this list of 
competencies was selected for reference over other lists established by other professional 
organizations such as the International Society of Performance Improvement (ISPI) or the 
Association for Talent Development (ATD). The IBSTPI organization does, however, focus on 
non-school setting competency development (Koszalka, Russ-Eft, & Reiser, 2013). The IBSTPI 
ID competency model in use by Army University, therefore, is the model referenced for the 
interview and coding schema developed for this study to determine practices of Army ISSs. 

Army Instructional System Specialist (ISS) Position Description 

The ID field has been in a state of constant development and change, moving in a myriad 
of directions and leading to a need for clearly delineated duties and tasks (Larson & Locke, 
2009; Raynis, 2018). In 2000 and 2016, researchers conducted extensive analyses of ID job 


announcements and associated competencies (Byun, 2000; Raynis, 2018). In both analyses, job 
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announcements were collected and “key responsibilities and qualifications for instructional 
designers in Corporate, Government/Military, Health, Higher Education, and Non-profit 
industries” were analyzed (Raynis, 2018, p. 166). Announcements were collected via headhunter 
sites, such as Monster, ATD, EDUCAUSE, Indeed, and Indeed health (Raynis, 2018) or via 
email dissemination by the Instructional Systems Technology (IST) Department at Indiana 
University (Byun, 2000). Results suggested that announcements for government/military ID 
professional competencies, such as design and development, communication and collaboration, 
assessment and evaluation, project management, and teaching and mentoring were below 4.4% 
(Raynis, 2018). In Byun (2000), Government/Military ID competencies were grouped with Non- 
Profit Organization (NPO) data but reported only NPO specific results. The results of these 
studies were inaccurate. The competencies did not accurately reflect the ID positions within the 
Government/Military sector for two reasons, which will be discussed next. 

First, job announcements were collected using the term Instructional Designer. 
Government/Military ID positions were titled ISS and therefore did not show in the job 
announcement sources that were used. Secondly, ISS positions were posted in the USAJobs 
announcement site, which was not one of the referenced sites of Raynis’ (2018) analysis. “The 
percentage of ID professionals reported as working in each career environment varies by the 
source cited...the discrepancy in figures between sources is often a result of the population and 
sample selected” (Larson & Lockee, 2004, p. 23). Therefore, it was important to have a general 
understanding of not only the design process and the ID competencies in use by Army 
organizations, but to also have an understanding of how the Army described the purpose and 


duties of the ISS position. 
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This section, therefore, provides a brief outline of the position description of an ISS at the 
GS 11 — 12 level. GS stands for General Schedule and is representative of a ranking order pay 
scale system for civilian personnel. GS 7-9 are considered interns. GS 11 and 12 are considered 
to be journeyman level. GS 13-15 are considered experts and are often in managerial or 
supervisory positions. They are considered capable and work independently. The work they 
complete is reviewed by supervisors for “suitability and effectiveness in meeting expected results 
in accordance with needs” (FASCLASS, 2005). 

Most ISSs occupy GS 11 and 12 positions. For the ISS position at this level, individuals 
are expected to apply the ADDIE process with the intent of providing educationally sound advice 
and guidance to all echelons of leadership about the theories and strategies of learning as well as 
training design and development. The ISS is expected to respond and adapt to information, 
procedures, and processes that change frequently while at the same time performing a variety of 
highly complex, intensive tasks which results in multiple instructional products which includes 
audit trail documentation of the design process as well as executable curriculum development 
courseware materials (FASCLASS, 2005). 


Table 3 


Position Duties: 1750 Instructional Systems Specialist GS-11 


Category Duty Description 
Product e Establishes & maintains historical audit trails for all actions. 
Manager e Prepares & presents briefings. 
e Writes studies & correspondence as required. 
e Uses initiative and judgment in applying & adapting broad educational 


principles, general administrative policies, & limited guidelines to the 
development & control of training Programs of Instruction (POIs). 

e Manages training development actions with other activities, organizations, & 
agencies. 

e Develops & recommends command position on assigned subject & areas. 

e Evaluates impact of new software & equipment in advance of initial training, 
doctrine, evaluation results, & long range trends. 

e Recommends cost effective training strategies. 


Coordinates work with higher headquarters, other education or training 
specialists, counterpart action officers, instructors, Subject Matter Experts 
(SMEs), TRADOC schools, & other appropriate agencies to plan, develop, 
& recommend solutions to training problems. 

Reviews, comments, & makes recommendations on education, training 
directives, & contracts for training development, evaluations, and reports. 


Analysis 
Team 
Member 


Acts as independent action officer or project officer, & advisor. 

Analyzes assigned projects to determine appropriate methodology, required 
research, required subject matter expert assistance, and need for educational 
surveys & related matters. 

Serves as the principal point of contact for design actions within the scope of 
assigned responsibility. 

Applies established criteria in selecting tasks for training & in 
recommending appropriate instructional setting. 

Collects data necessary to support design projects. 

Reviews & analyzes internal and external feedback & applies results to 
determine adequacy of task analysis and training program documentation. 
Evaluates adequacy of material collected & initiates requests for additional 
data. 


Developer 


Develops, coordinates, recommends approval, & makes changes to: 
o Programs of Instruction (POD, 
Individual Training Plans (ITPs), 
Course Administrative Data (CADs), and 
Course Management Plans (CMPs) for 
Training course materials (i.e. presentations, lesson plans, student 
handouts, assessments) 
Write regulations, pamphlets, procedure guides, policy documents, & SOPs. 


oO00 0 


Evaluator 


Trainer 


Reviews, coordinates, or initiates surveys & reports to appraise impact on 
assigned area of operations & programs. 

Prepares correspondence in support of assigned projects. 

Evaluates & provides guidance & direction a course development proposals, 
training device requirements, new training media, course design, & training 
materials. 

Determines methods of solving training problems pertaining to the 
implementation of training concepts, techniques and procedures. 

Trains, advises & evaluates instructor personnel in the development of 
training materials. Ensures compliance with TRADOC regulations, 
USAAVNC guidance & educational soundness. Observes and evaluates 
instructor performance & recommends corrective actions as needed. Detects 
needs for & submits recommendations concerning modifications of all 
portions of the training programs including innovative or advanced training 
techniques, approaches to subject matter, sound educational principles & 
procedures & the best accepted tenets of instructional technology. 


Note: Adapted from FASCLASS, (2005). 
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Education vs. Experience 

An extensive review of the literature on ID practice and perception strongly suggests a 
desire within the profession to orient ID education toward practical application readiness rather 
than theoretical understanding. The general perception of academic programs is that if a student 
knows and understands various design theories and models, then translating them into practice is 
easy and natural. Research in the field of practice, however, reports something quite different 
(Christensen & Osguthorpe, 2004; Cox & Osguthorpe, 2003; Osguthorpe & Osguthorpe, 2007; 
Rowland, 1993; Tracey & Boling, 2014). 

Practitioners with experience within the profession rarely utilize design theories as 
traditionally taught. They routinely modify and augment principles and techniques of one theory 
with principles and techniques of other design theories, as well as theories outside of the teaching 
of instructional design. Some of these supplemental theories include theories from educational 
psychology, decision-making, systems, and change management theories from business and 
management educational arenas. 

The same literature also indicates that few students are prepared for realistic entry into 
the workforce as instructional designers upon graduation. Academic programs in IST or ID 
reportedly focus heavily on a strong foundational knowledge of various instructional design 
theories and models. In contrast, those practicing ID report a significant lack of strict adherence 
to said theories and models. 

Two themes emerged during a review of the practice of ID literature. Theme one 
describes the difference between academic learning and design practice. Theme two describes 
the characteristic differences between novice and expert designers. The following section will 


elucidate upon these themes. 
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Theme 1: Academia vs. Practice 


There has been a recurring question in literature about the practice of ID: What do IDers 
actually do (Christensen & Osguthorpe, 2004; Cox & Osguthorpe, 2003; Dennen & Spector, 
2007; Gray et al., 2015; Larson & Lockee, 2009; Osguthorpe & Osguthorpe, 2007; Parrish, 
2009; Rowland, 1993; Tracey & Boling, 2014; Williams et al., 2011)? Researchers have been 
asking this question because they want to know whether or not academic programs for 
instructional design are realistically preparing instructional design students who are able and 
ready to practice the art of instructional design in the workforce. Additionally, researchers have 
been studying whether or not academic programs are teaching the right information correctly to 
prepare students for how instructional design is practiced in the workforce. 

Overall, the literature on the practice and perception of ID presents several ideas 
regarding the profession and practice of instructional design. The first idea is that there are no 
academic requirements to enter into the profession of instructional design (Paquette, 2014). In 
fact, research suggests that a large percentage of those professionals with the title [Der have no 
educational background in the field. Schwier et al. (2004) state, “Instructional Designers come 
from many professional walks of life, and there doesn’t seem to be central rallying post for 
them” (p. 81). 

The second idea is that most academic programs in instructional design are at a master’s 
level or higher (Dennen & Spector, 2007; Tracey & Hutchinson, 2013). There are very few 
undergraduate programs available for the profession. Additionally, those administrators in the 
limited undergraduate programs assume that bachelor’s degree students have acquired the same 
level of education as those completing a master’s degree in ID (Cox & Osguthorpe, 2003). 

The third idea is that there is a distinct and significant difference between what ID 


students are taught and how they practice once they have graduated and enter the 
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workplace (Christensen & Osguthorpe, 2004; Cox & Osguthorpe, 2003; Dennen & Spector, 
2007; Gray et al., 2015; Larson & Lockee, 2009; Osguthorpe & Osguthorpe, 2007; Parrish, 
2009; Rowland, 1993; Tracey & Boling, 2014; Williams et al., 2011). Researchers report that 
educational programs focus heavily on the knowledge of various instructional design theories 
and their associated models. The emphasis is primarily oriented toward understanding the 
historical, philosophical, psychological, sociological, and/or economical influences that comprise 
each theory. Therefore, the importance is on understanding how the model is intended to work 
and what the expected outcomes are proposed to be. 

ID students typically learn models in a linear fashion even though literature strongly 
suggests the opposite is true. ID practice is often disjointed and irregular in comparison to model 
structures. Schwier et al. (2004) states, “much of the extensive work describing theoretical 
models of ID has not been drawn from the practice of the designer and consequently 
instructional design theory is not grounded in practice” (p. 64). Additionally, Tracey and Boling 
report that research conducted in 2000 indicates that [Ders are practicing significantly differently 
than how they have been taught to perform as Instructional Designers in academic programs. 
Professionals hired to fill ID positions are not entering the workforce ready to perform tasks 
identified as essential for their organization (Tracey & Boling, 2014). 

In order to establish some semblance of structure within the field, the IBSTPI developed 
a list of competencies for instructional designers that have been well recognized throughout the 
academic and corporate arenas since 1983. These competencies were only recently incorporated 
into military training organizations in 2017. Some IBSTPI competencies, however, have not 


been frequently used in practice due to of lack of time and resources, control over decision- 
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making, designer perceptions of a task, underlying philosophical beliefs, and designer expertise 
(Dennen & Spector, 2007; Larson & Lockee, 2009; Tracey & Boling, 2014). 

Research shows that those practicing instructional design rarely use entire theories and 
models during their work and academic programs fail to account for the complexities and 
constraints of workplace practice (Rowland, 1993). The literature on ID practices contends that 
ID educational programs should consider studio oriented training that provides ID students with 
the opportunity to engage in ID application in order to experience the intricacies of practice 
(Dennen & Spector, 2007). Programs should also consider how to translate knowledge of 
theories and model to practice within non-design thinking environments. Non-design thinking 
environments are defined as environments outside of the instructional design departments 
(Tracey & Hutchison, 2013). Reminiscent of the work performance expectations identified for 
ISSs earlier in this chapter (FASCLASS, 2005), Tracey and Hutchinson (2013) strongly 
encourage IDers to be actively cognizant of non-design thinking throughout their organizations. 
This means that IDers should anticipate the need for patience and flexibility; prepare to offer 
multiple solutions to presented problems; adapt processes and procedures based on data, as well 
as desired outcomes; and communicate all of the above in a clear and understandable manner. 

IDers need to be flexible and adaptable within their field. There are too many variables in 
an ever-changing environment for professionals in this field to remain rigid and without proper 
and applicable education. This may be inherently difficult for ISSs working in government or 
military training environments that base their operations on hierarchical organization, doctrine, 


and regulation. 


27 


Theme 2: Industry Novice vs. Expert Task Performance 

Gibbons (2003) states, “instructional designs can indeed be conceived of as multiple 
layers of decision making with respect to different sets of design constructs, and we find a rough 
correspondence between the layers and the phases of designer thinking” (p. 23). The 
aforementioned layers of thought and the evolution of understanding impacts ID practice and can 
work to distinguish novice from expert. The perception of theory and application of instructional 
design varies greatly from novice to expert designers. 

Novices work diligently to understand the problem before considering solutions, while 
experts use solution ideas to help clarify the problem (Tracey & Boling, 2014). Without years of 
practical experience to guide them, novices move from model to design thinking by spending 
more time defining problems rather than innovating solutions (Tracey & Hutchison, 2013). 
Novices need mentorship from experienced practitioners and experiential learning opportunities 
in order to recognize the tools and techniques of applying theories and principles to real-world 
projects. Mentorship should begin at the academic level and continue into their career (Tracey & 
Boling, 2014). Without this guidance, novices are quick to believe the information they are 
given, and without questioning it, move to solution innovation without proper and reliable 
information (Rowland, 1992). 

The innovation process of an expert [Der varies greatly from that of novices. In contrast, 
experts question the validity of provided documentation; refer to model constructs; hypothesize 
Subject Matter Expert (SME) and peer interactions; include non-training solution alternatives; 
and do not need to return to problem identification when devising solutions (Rowland, 1992). 
Additionally, to those with years in the field, the process of instructional design is a learning 


experience, which includes collaboration, conversation, and self-reflection (Osguthorpe & 
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Osguthorpe, 2007, p. 74). Reflective skills have a positive impact on instructional design 
application with time, practice, and guidance (Tracey & Hutchison, 2013). Research reports that 
expert instructional designers understand and practice self-reflection by comparing their beliefs 
against theories in the field (Osguthorpe & Osguthorpe, 2007). By doing so, they hold 
themselves to a high moral and professional standard, something that is referred to as the 
“conscience of craft” (Osguthorpe & Osguthorpe, 2007, p. 21). 

Novices in the field of ID lack the experience, understanding, and self-reflection of 
experts. They can eventually, however, perform on the same level as expert IDers if they are 
taught to do so (Tracey & Boling, 2014). Researchers believe that with mentoring, an expert can 
help develop this mindset by “not only thinking about models and theories, but about the 
assumptions and beliefs that give rise to those models and theories” (Osguthorpe & Osguthorpe, 
2007, p. 21). Reflection can assist novices in overcoming theoretical knowledge and transform 
their understanding to a more applicable and solution focused process. This would allow novices 
to develop the practical and applicable innovation of expert instructional designers. 

Summary 

This chapter discussed the procedures of reviewing instructional design literature, the 
history and formation of ID, and different definitions of the role by several researchers. I 
examined the use of the ADDIE process in both the instructional design field as a whole, as well 
as its application in a military setting. The ID IBSTPI competencies were also discussed. I noted 
how competencies seek to clarify expectations within the field and were updated over time to 
compensate for the fluidity of instructional design. Gaps were acknowledged within the literature 
on the practice of ID, where ID is being omitted simply due to lack of understanding of the true 


depth and complexity of the field. Education is also proving to be inconsistent for those within 
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the ID workforce, where what is being taught is not being practiced. Understanding the 
complexity and constantly changing work environment, as well as need for practical application 
in the field, and better understanding of instructional designers and what they do will allow for a 


better and more stable educational pathway for those who practice now, and in the future. 
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Chapter Three: Research Design 

This study was conducted using a descriptive case study research design. This researched 
intended to study and provide detailed insight into the real-world phenomena of instructional 
design as it occurs within a specific military environment. 

There were three research questions, which guided the study: 

RQ1: What type of education and/or training have Instructional Systems Specialist (ISS) 
practitioners received as Instructional Designers (IDers)? 

RQ2: What International Board of Standards for Training, Performance, and Instruction 
(IBSTPI) Instructional Design (ID) tasks are performed most regularly by Instructional Systems 
Specialists (ISSs)? 

RQ3: What is the perceived level of expertise requisite for performance of Instructional 
Design (ID) tasks by Instructional Systems Specialists (ISSs)? 

Research Design 

Qualitative data were collected via interviews and follow on interview questions that 
were subsequently developed specific to the responses provided by the interviewees. The 
intentions of the interview questions were to answer this study’s research questions and to 
establish a foundational link to the themes of “Active vs. Practice” and “Novice vs. Expert” 
gleaned from the literature on the practice and perceptions of Instructional Design (ID). 
Interviewees were bound by the following sampling criteria perimeters: Interviewees filled the 
civilian position of Instructional System Specialist (ISS) at a GS-11 or GS-12 level, and they 
were employed at the United States Army Aviation Center of Excellence (USAACE) at Fort 
Rucker, Alabama during the time of the interviews. ISS personnel selected to participate in the 


interviews covered a civilian service period of 39 years, from 1980 to 2019. The years of 
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experience of interviewed ISSs ranged from one to 36 years. Results gleaned from the data 
gathered and inferences made were only representative of that population. Yin (2017) purported 
that descriptive case study design could be used to, “present rarely encountered situations or 
situations not normally accessible to researchers” (p. 264). In this case, data was gathered within 
military training environments, which were not normally presented within the literature targeting 
instructional design practices. 

Reflective of the research design present in the Instructional Design practice literature, 
this study also modeled the research of Schwier et al. (2005). In contrast to other types of 
research design present in the literature on ID practice, this study allowed for an inductive 
analytical process. This meant that data was collected and reviewed; patterns were identified and 
coded against existing model themes; and then theories emerged. Much of the literature on the 
practice of ID located for this study did not explicitly describe the research design used to collect 
and analyze its data. 

The literature suggests that many studies on ID practices and perceptions, however, are 
representative of this inductive process. In most cases, research depicts data collection via survey 
and/or interview initially. Theories, conclusions, and questions for further contemplation are then 
devised from the resultant data. The ID practices literature does not usually depict a deductive 
analytical process where a hypothesis is initially proposed and then proven by the analytical data. 

Research Participants 

Participants in this study were Army civilian training personnel within the United States 
Army Aviation Center of Excellence (USAACE) at Fort Rucker, Alabama. Every civilian 
position was housed under a career program. The career program for ISS was Career Program 32 


(CP-32). The CP-32 designates the civilian training and war-fighting developers. The program is 
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further divided into General Support (GS) series of work, also designated by number. The CP-32 
series included in this study were Series 1701, Series 1712, and Series 1750. CP-32 employees in 
GS-1701 General Education and Training positions advise, administer, supervise, and/or perform 
work in the field of education and training. 

CP-32 employees in GS-1712, or Training Specialist positions, are involved in the direct 
delivery of instruction or services. Training Specialists (Series 1712) tend to be prior military 
personnel with professional experience in the field being instructed (U.S. Department of the 
Army, 2017b). In contrast, CP-32 employees in GS-1750, or Instructional Systems Specialist, 
positions perform professional work in training. This particular group was the focus of this 
research. Many of the 1750s serve as instructors, supervisors, administrators, and managers in 
academic and technical-vocational programs. Others provide professional educational principles 
and theory in the analysis, design, development, implementation, and evaluation of training 
programs and products. Employees classified in the GS-1750 series must have completed a full 
four-year course of study leading to a bachelor’s degree or higher, which included, or is 
supplemented by, at least 24 semester hours in education. The 24 credit-hour coursework must 
include study in learning theory, psychology of learning, educational psychology, instructional 
design practices, educational evaluation, instructional product development, or instructional 
technology (U.S. Department of the Army, 2017b). This research focused specifically on those 
hired into ISS positions. 

Sampling and Selection Criteria 

Participants were selected in accordance with the samplings criteria listed in the research 

design section in order to ensure a thorough representation of the convenience population. The 


criteria with the greatest impact on the selection of participants was the civilian service date. This 
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date indicated how long someone had been working as an ISS within civilian service. Civilian 
service dates for GS11-12 level ISSs at USAACE spanned from 1980 to 2019. As of 30 Jan 
2019, USAACE employed 48 ISSs. Eight of those 48 were in GS-13 or higher positions and 
fulfilled managerial or supervisory positions. 

At the time of the data collection interviews, two of the remaining 40 positions were 
vacant, leaving 38 ISSs available to select for participation in this study. Civilian service dates 
and associated personnel were divided up by decade. ISS personnel were then randomly selected 
by percentage from each decade in order to ensure the most accurate representation of ISS 
experience reflected in this study. 

There were four ISSs with civilian service dates beginning in the 1980s. One of four was 
randomly selected to participate in the interview. That selected interviewee’s service 
computation date was 1984 and represented 36 years of ISS experience. There were three ISSs 
with civilian service dates beginning in the 1990s. One of three was randomly selected to 
participate in the interview. That selected interviewee’s service computation date was 1999 and 
represented 21 years of ISS experience. There were 16 ISSs with civilian service dates beginning 
in the 2000s. Nine of the 16 were randomly selected to participate in the interview. The selected 
interviewees’ service computation dates ranged from 2004 to 2009 and represented 11 to 16 
years of ISS experience. There were 15 ISSs with civilian service dates beginning in the 2010s. 
Eight of those 15 were randomly selected to participate in the interview. The selected 
interviewees’ service computation dates ranged from 2010 to 2019 and represented one to 10 
years of ISS experience. Interviews represented 50% of the number of GS-1750 11-12 ISSs at 
USAACE. Nineteen ISSs were invited to participate in the interview for data collection. 


Seventeen ISSs participated in the interviews. 
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Study Setting 

I solicited all 19 identified USAACE instructional systems specialists in order to gain a 
perspective of instructional design practice applicable in U.S. Army training organizations. Each 
ISS received an email inviting them to participate in a face-to-face interview and follow up 
questions based on interview responses (see Appendix B). The email included a personal 
introduction, a description of the study, a request to participate, and a copy of the consent form 
that they would be asked to review and sign at the time of the interview. I managed the local CP- 
32 program at the time of the study. I believed that 17 of the 19 respondents agreed to the email 
request to participate because they knew and trusted me. 

Data Sources 

For this study, I developed an interview protocol that was distributed to the selected 
USAACE ISS population. The interview addressed all three research questions. Interview 
respondents were informed that follow up interview questions would be conducted in order to 
gather more in depth understanding of thematic interview responses. Respondents were advised 
that follow up questions would occur spontaneously during the interview process and also 
following the conclusion of the interview as themes emerged during data analysis. The purpose 
of the interview was discussed; the consent form was thoroughly reviewed and signed; and the 
respondent was asked permission to audio-record their respective interviews prior to the start of 
each interview. 
Interview Protocol 

An interview protocol was developed to provide a baseline of common questions that 
were asked of all interview participants. In addition to the question set, the interview protocol 


included: 1) basic information about the study and the intended use of interview data; 
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confidentiality; 2) methods available for interviewing (e.g., in-person, telephonic, electronic) in 
order of research preference; and 3) an informed consent form. 
Interview Section One 

RQ1: What type of education and/or training have Instructional Systems Specialist (ISS) 
practitioners received in Instructional Design (ID)? 

The literature review for this study highlighted that most IDers come to the profession 
from varying careers and educational backgrounds. The first portion of the interview questions 
asked about the level of degree earned in ID. It asked about the date in which that degree was 
achieved. In addition to formal education, these questions asked respondents to identify which, if 
any, of the Army’s Faculty and Staff courses have been completed and when. The Faculty and 
Staff courses were developed by Training and Doctrine Command (TRADOC) entities and 
provided to all other installation Faculty and Staff sections for instruction. The following two 2- 
week courses were the only required courses for all personnel—amilitary or civilian—assigned as 
Army instructors or training developers: the Common Faculty Development—Instructor Course 
(CFD-IC), and the Common Faculty Development—Developer Course (CFD-DC). Both courses 
were re-designed as recently as 2018. Much like the ID field itself, the Faculty and Staff 
instructor and design related courses have gone through numerous changes within the last 10 
years in order to remain aligned with industry standards, practices, theories, and innovations. For 
example, the Army instructor training course has iteratively been redesigned and been called: the 
Instructor Training Course (ITC), the Total Army Instructor Training Course (TAITC); the Army 
Basic Instructor Course (ABIC); The Foundational Instructor Facilitator Course (FIFC); and now 


the Common Faculty Development—Instructor Course (CFD-IC). 
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Types and dates of education and training received spoke to the currency and relevance 
of practitioner knowledge regarding instructional design principles, as well as their ability to 
practically apply new ID techniques and strategies. 

Interview Section Two 

RQ2: What International Board of Standards for Training, Performance, and Instruction 
(IBSTPI) Instructional Design (ID) tasks are performed most regularly Instructional Systems 
Specialists (ISSs)? RQ3: What is the perceived level of expertise requisite for performance of 
Instructional Design (ID) tasks by Instructional Systems Specialists (ISS)? 

Interview questions in the second portion asked respondents to indicate: 1) what tasks 
they perform as an ISS; 2) what skills are used as an ISS; 3) what tasks/skills used are considered 
difficult or easy to perform; and 4) the perceived degree of difficulty of task performance, as 
well as 5) the perceived importance and expertise required for design tasks. For interview data 
coding purposes, expertise options were defined as Essential, Advanced, or Managerial (IBSTPI, 
2020). An essential level of skill expertise must be exhibited by all personnel within an ID 
position. An advanced level of skill expertise must be exhibited by experienced personnel within 
an ID position. A managerial level of skill expertise must be exhibited by those in supervisory ID 
positions. 

The TRADOC Regulation 350-70 dictates the use of the Analysis, Design, Development, 
Implementation, and Evaluations (ADDIE) Process by those working in Army training sites. The 
Army University (ArmyU), an organizational off shoot of TRADOC, recently began to reference 
the IBSTPI ID competencies as part of their design consideration. Unfortunately, this 
information had not been disseminated well to the ISS populace. Kelly (2016) conducted an 


extensive review of instructional design competencies from multiple professional organizations 
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to include IBSTPI. Kelly provided a list of IBSTPI competency tasks by phase of the ADDIE 
process. The second portion of this study’s interview protocol utilized that competency list by 
phase in comparison to competencies or task performance identified during ISS interviews. 


Table 4 
Instructional Design Competencies for Instructional Designers 


ADDIE PHASE IBSTPI Competency Tasks 


Analysis Analyze content from SMEs 

* Analyze existing instructional products 

* Analyze various types and sources of content 

* Conduct a needs assessment 

* Describe the learning or performance problem 

¢ Determine subordinate and prerequisite knowledge and skills 

¢ Determine the breadth and depth of the instructional content 

* Determine the cause of problem 

* Determine the impact of organizations’ characteristics on the suggested 
solution 

* Estimate costs and benefits for the solutions 

* Identify and describe target population and environmental characteristics 
* Identify needs among stakeholders 

* Identify the infrastructure 

* Identify the scope of content for instruction 

* Present needs assessment report 

* Provide instructional and non-instructional solutions 

* Select and use analysis techniques for determining instructional content 
* Use learners’ data to design instruction 


Design Accommodate individual factors in learning 

¢ Apply appropriate interaction design and interactive learning principles 
* Apply appropriate motivation principles 

* Describe the rationale for designing instructional process 

* Design instructional interventions 

¢ Design learning assessment 

* Determine the overall scope of instruction 

* Identify and sequence instructional goals 

* Identify and sequence learning objectives and outcomes 

* Identify appropriate instructional strategies 

* Modify the instructional design process 

* Plan non-instructional interventions 

* Select or create the appropriate instructional design process 

* Use appropriate visual design principles 

Develop Deliver instructional materials in various formats 

* Develop instructional materials 

* Develop materials which align with analysis results and instructional 
design process 
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* Select and modify existing instructional materials 
Implement Plan for implementation of the intervention 

¢ Plan for dissemination of the intervention 

* Plan for diffusion of the intervention 


Evaluate Design evaluation plans 

* Conduct formative evaluation plans 

* Conduct summative evaluation plans 

* Revise instructional and non-instructional solutions 


Note: Adapted from Kelly (2016). 


Interview protocol development validity. 

Validation of the interview protocol was important for credibility of the research. The 
interview protocol designed for this study went through a three-part validation process. Part one 
validation consisted of a drafted instrument of items being sent to three expert instructional 
designers for review and comment. Expertise in this case was determined by two criteria. First, 
experts were professionals who had been working actively within the field for 10 or more years. 
Second, experts had experience in either teaching ID principles and/or in designing curriculum. 
Two of the experts were Army IDers. One expert was a tenured professor at Indiana University 
within the Instructional Systems Technology Department in the School of Education. 

The draft items were delivered to the three experts in a word document. Comments were 
adjudicated and modifications were made to the item structures. Lastly, the interview protocol 
was provided in the same manner to five personnel representative of the intended target 
population who were not selected to participate in the study based on the sampling criteria. The 
target audience representatives were given a word document onto which they were instructed to 


readily make comments. 


Table 5 


Interview Protocol Reviewer Validation Comments 


Reviewers 


Validation comments of Draft 


Expert 


Looks Good 

If you ask about the CFD-DC (Developers Course), we aren’t 
going to start teaching that until early next year. You might 
want to consider adding FTDC or replace CFD-DC with FTDC. 


Note: FTDC is the Foundational Training Developer Course and 
was the precursor to the newly redesigned CFD-DC. 


Target Audience 


Representatives 


Questions are very thorough 
Seems on point 
Do master’s degrees have minors? 


Reviewers 


Expert 


Target Audience 


Representatives 


Validation comments of Instrument development 


Q1: This is confusing — “graduate” degrees are anything masters 
and above. And I have no idea what a “post graduate” degree is. 
So eliminate those options and say “masters” “doctorate” 
“other.” 

“If you achieved a graduate degree” — change to “masters 
degree” 

I like the ideas embedded in the interview! 

Looks like your interview collects qualitative and quantitative 
information. 

Part 2, you're asking for a value judgment; "In your opinion, 
how important are..." Is this informed by experience, evaluation, 
unit effectiveness, etc.? Point here is how do you ensure 
reliability? Otherwise, looks like same questions. 

Part 2 is tricky. How do you define level of expertise for your 
audience — are these categories part of shared knowledge and 
understanding within the community of practice? 

I think this looks pretty good. 


Interview Procedures 


fe 


Due to the nature of this study, the interview for this study was semi-structured with a set 


of pre-planned leadoff questions followed by further, unplanned questions to obtain clarification 


on what is discussed by the interviewee. Following the validation of the interview protocol, I 
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sent 19 individual emails to those individuals who were selected to participate in this research. 
Emails were sent to each potential interviewee to better ensure confidentiality. As previously 
mentioned, email invitations included the description of the research as well as an advance copy 
of the consent form for their review and questions. 

Interviewees were given two weeks to send their response in writing with their agreement 
to participate. They were asked to reply in response to the email invitation to provide additional 
indication of their willingness to participate prior to the review and signature of the consent 
form, which occurred at the start of each interview. As interviewees replied, appointments were 
scheduled for one-hour sessions during the months of April and May 2019. Interviews occurred 
during official work hours and were scheduled at the convenience of the interviewee. 

All interviews took place in the 110“ Aviation Brigade Headquarters second floor 
conference room. This conference room was chosen because it was located outside of my 
immediate workspace and was a neutral, generally unfamiliar location to most of the 
interviewees. All interviews followed the set protocol. Interviews began with a personal greeting. 
There was a re-introduction to the purpose of the study; a review of the expected confidentiality; 
a review of the consent form; a request to sign the consent form; and a request for permission to 
audio-record each interview. 

Interviews were recorded and interviewees were informed that all audio-recordings 
would be stored within the application directly to my personal device. Each interview began with 
the same Interview Protocol questions (See Appendix A). Each interviewee was asked each 
question annotated on the Interview Protocol. Follow-on questions were asked throughout the 
interview process as needed to gain more insight into interviewee responses. Interviews lasted 


between 31 and 71 minutes. The average interview length lasted 45 minutes. At the conclusion 
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of the interview, each respondent was informed that the interview would be transcribed, 
reviewed for accuracy by me, and then sent to them for their review for accuracy. 

An interview transcript was provided to each interviewee for review and verifications that 
information accurately reflected interviewee thoughts, opinions, comments, etc. Corrections to 
transcribed comments were made prior to any research review or coding. Respondents were 
informed that they were to review their respective interview transcript for errors and/or 
elaboration on any of their comments. Of the 17 respondent reviewed interviews, five were 
returned with comments or corrections. I sent the remaining 12 interviewees an email asking 
them to confirm their approval of their interview transcripts. All replied in the affirmative. 
Codes were initially derived from pre-established categories reflecting IBSTPI competencies 
within each of the ADDIE design process phases. 

Interview Data Analysis 

Interview responses were reviewed and coded. Data emerged from the interviews and 
were fitted the to the pre-established five IBSTPI categories: 1) Professional Foundations; 2) 
Planning and Analysis; 3) Design and Development; 4) Evaluation and Implementation; and 5) 
Management. Interviews were transcribed using the Rev application for voice recording, which 
provides a transcription option. First, I reviewed the transcriptions for emerging thematic 
information. Topical codes (i.e., descriptive terms or phrases) were aligned with tasks 
represented by phases of the IBSTPI competency categories as presented in Table 2. 

In order to validate emerging codes matching IBSTPI categories, one of the two Army ID 
experts used to validate the interview protocol was asked to review interview transcripts and 


code the data from their perspectives. 
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The IBSTPI categories were provided to each expert to establish their coding baseline but 
were also asked to identify any emerging themes that they saw within the data as well. The 
reviewing ID expert and I added comments to the Interview Quote Log. I established the 
Interview Quote Log to capture transcription quotes, by interviewee, that addressed each 
research question: educational background, IBSTPI ID categories, and expertise. The transcripts 
were first reviewed independently by me and entered into the Interview Quote Log. Then I 
shared the transcripts and Interview Quote Log with the ID Subject Matter Expert (SME). The 
ID SME was asked to annotate any disagreement or questions with initial coding. Comments 
identified both reviewer and my thoughts on IBSTPI categories relative to respondent comments 
as well as other thematic data that caught our attention. Once the expert had completed the 
review and coding, we met to discuss and adjudicate feedback. Coded information was decided 
and finalized, by category or theme, within an Excel document. Each code disagreement or 
question was reviewed and discussed until both reviewers agreed on the appropriate quote and 
category matching. 

Potential Limitations of the Study 

Given the structure of this research design and the convenience sample, findings and 
discussions are applicable only to the study’s population. This research is applicable for Army 
UD practices. The findings are not generalizable to the wider profession of ID. 

Another potential limitation of this study was that researchers had previously built their 
own survey instruments to gather data. The same held true in this study. Due to the specific 
nature of this proposed research and the unavailability of pre-existing surveys, acquisition and 
use of previously developed and validated data collection tools were not possible. I believe that 


use of a survey, questionnaire, or interview instruments used in past research would increase the 
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credibility, as well as the comparability of the data collected in this study. This would also 
provide a more concrete link to the existing literature. 

Due to my primary access to Army personnel, a second potential limitation was that this 
study will reflect the Army training organization only. It will not represent all military training 
branches, which includes the Air Force, Navy, Marines, and Coast Guard entities. Therefore, the 
scope of this study will represent only a portion of the greater military and government 
workforce. 

As I was employed as an Instructional System Specialist GS-13 within the United States 
Army Aviation Center of Excellence and also acted as the career program manager for 
instructional personnel at the time of this study, a third potential limitation was anticipated. As 
the career program manager, I was familiar with all selected interview participants. The 
immediate working relationships increased exposure to the researcher and the study itself. It may 
also have increased the possibility of response bias. Interviewees were advised that participation 
was completely voluntary in order to mitigate any potential bias. 

Summary 

This chapter provided a description of the descriptive case study research design that best 
suited the environment, situation, and population encompassed by this study. It described the 
target population and the sampling criteria by which participants were selected. This chapter 
detailed the interview process and procedures used to ensure neutrality and well and confidence 


in confidentiality. 
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Chapter Four: Results and Conclusions 
This chapter presents the analytical results for the following three research questions: 
RQ1: What type of education and/or training have Instructional Systems Specialists 
(ISS) practitioners received as Instructional Designers (IDers)? 
RQ2: What International Board of Standards for Teaching, Performance, and Instruction 
(IBSTPI) Instructional Design (ID) tasks are performed more regularly by Instructional 
Systems Specialists (ISSs)? 
RQ3: What is the perceived level of expertise requisite for performance of Instructional 
Design (ID) tasks by Instructional Systems Specialists (ISSs)? 
Description of the Researcher 
I began my career in ID in 2001 when I entered into an instructional design and 
Development program at the University of South Alabama. At that time, I had never heard of 
Instructional Design as a profession. I was introduced to the academic program by a friend. Both 
of our undergraduate and graduate experience were in psychology and counseling. I found that 
often within our counseling sessions I was acting as a mentor or life coach, teaching people how 
to identify problems, analyze situations and information, select approaches to addressing those 
problems, and evaluate progress. I planned counseling sessions as individualized training plans 
with motivations, activities, appropriate counseling methods selection, etc. In short, it seemed 
like a good fit and one that would advance my skill set while also expanding my career options. 
This experience seems to reiterate Paquette’s (2014) and Schwier’s (2004) assertions that people 
come the ID profession through varying career and educational paths. 
In 2003, I began my career as an Instructional System Specialists (ISSs) within the Army. 


In 2005, I became an adjunct professor of ID principles within the Workforce Education and 
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Development program through Southern Illinois University. I learned more about instructional 
design practices as an educator within an instructional design program. I became more adept at 
practicing and understanding the concepts that I had learned. 

I designed courses for the Army as a member of Staff and Faculty and then as an ISS, 
working on Flight course materials for the United Stated Army Aviation Center of Excellence 
(USAACE). During this experience, I also became aware of the overall perceptions of ISS within 
that environment, both by those who led the professionals and by those in the profession. I 
wanted to see ISSs used to their full potential to improve the quality of training provided, as well 
as address the resourcing needs of the Army. I was motivated by the desire to understand how to 
best leverage a professional skill set so that both product and process work toward the greater 
good of the organization. This, ultimately, resulted in a well-trained soldier performing his duties 
without failure or injury to self or equipment. 

Over the course of the past seventeen years, my passion for the profession has grown as 
has my frustration in the ability to fully perform as a designer. I have adopted a drive for 
organizations—particularly the government/military because of the nature of what they do—to 
understand what the profession actually is, as well as the broad spectrum of skill capabilities 
possible within the profession. In this research, this passion has motivated me to expand the 
knowledge of instructional design practices and perceptions in military training environments. It 
was my experience within the field that comprised my own perceptions, which I had to 
consistently keep in mind when reviewing the opinion of those interviewed for this study. 

Participant Demographics 
The interviewees who participated in this study were all ISSs in the job series 1750 and 


employed as civilians at the United States Army Aviation Center of Excellence (USAACE), Fort 
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Rucker, Alabama from 1984 to the present. Years of ISS experience ranged from one to 36 
years. Civilian Service Computation Dates (SCDs) and persons filling ISS positions were 
provided by the local Civilian Personnel Advocacy Center (CPAC) upon request prior to data 
collection. Educational background demographics were the result of direct questions specific to 
research question number one. Military experience demographics were gathered when it was 
provided during the interview as part of a response. When military experience was not provided 
it was asked of interviewees during follow up interview questioning. 


Figure 2 


Demographic Information by Interviewee 


Interview | Gender | Civilian | Emply | Highest Yr Degree Subject Mil | Rank/Yrs | Branch Primary Job 
# Emply | years | Degree | Degree Exp Responsibility 
Date Achieved Y/N 
1 M 2018 1 Doc 2003 | Education Y | Enlisted/4 | F&S | TD- Implementation 
Technology 
2|_ M 2009 10 BS 2010 | Business Admin Y_| Enlisted/10 | F&S | TD- Implementation 
3 F 2019 >1 BS 2009 _| Secondary Education | Y | Enlisted/8 | FTIB_ | TD—Design 
4 F 2009 10 MS 2005 | Education N FTB_ | TD-Analysis & 
Technology Development 
5 F 2006 13 MS 2002 | Education N ETB | TD- Analysis 
Technology &Development 
6 M 2016 3 MS 2011 ‘| Education Y | Enlisted/30 | ETB | TD-Analysis & 
Development 
7| M 2007 12 MS 2007 | Instructional Y | Enlisted/24 | ETB | TD—Analysis & 
Technology Development 
8| M 2018 >1 BS 2018 _| Aeronautics Y_| Enlisted/26 | F&S | TD—Implementation 
9 F 2008 11 MS 2013 | Adult Education Y | Enlisted/11 | Collecti | TD—- Analysis & 
ve _| Development 
10} M 1999 20 AS 1993 | Education Y | Enlisted/12 | OTB | TD-Analysis & 
Development 
ll F 1984 35 MS 2011 | Post-Secondary N ETB | TD-Analysis & 
Education Development 
12; M 2007 12 MS 2016 | Education Y | Enlisted/24 | F&S | TD—Implementation 
Technology 
13| M 2004 15 BS 2002 _| Elementary Education | _N CRC_| TD—Development 
14 F 2006 13 MS 2013 | Adult Education Y Enlisted/6 F&S | TD-Design and 
Development 
15 M 2010 9 MS 2001 | Aerospace Education Y Enlisted/1 ETB | TD-Analysis & 
Development 
16 M 2011 8 MS 2013 | Education Y | Enlisted/24 | QAO | TD-Evaluation 
17 M 2015 4 MS 2015 | E-Learning &Higher Y Officer'22 | QAO | TD- Evaluation 
Learning 


Of the 17 ISSs interviewed, six were female and 11 were male. Four had no military 


background while 13 had military experience. Of the 13 with military experience, 12 were within 
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the enlisted ranks and one was within the officer ranks. For purposes of this study, rank was not a 
consideration for discussion but may be considered for future research. 

All of those participants who were considered career military were male. Non-career 
military persons were split between males (4) and females (3). Due to the 10-year gap in years of 
military experience, this study defined career military to be those who serviced as active duty 
military personnel for twenty years or more. Non-career military interviewees were defined as 
those not having serviced as active duty military or having served as active duty military 
personnel for 12 years or less. There was also a split between career (6) and non-career military 
(7) persons who participated in this study. 


Table 6 


Demographic Information by Research Question 


Category Interviewee 
Gender 
-Male (/1) 1, 2, 6, 7, 8, 10, 12, 13, 15, 16, 17 
-Female (6) 3,4, 5,9, 11, 14 
Military Experience 
No Military Experience (4) 4,5, 11, 13 
Military Experience (13) 1, 2, 3, 6, 7, 8, 9, 10, 12, 14, 15, 16, 17 
-Enlisted (12) 1, 2, 3, 6, 7, 8, 9, 10, 12, 14, 15, 16 
-Officer (J) 17 
-Career Military (>20 yrs) (6) 6, 7, 8, 12, 16, 17 
-Non Career Military (<l3yrs) (7) 1, 2, 3,9, 10, 14, 1 
Educational Background 
Instructional Design Degree (5) 1,4, 5,7, 12 
Adult Education Degree (2) 9, 14 
Education Degree (Elem, Second, Post, General) 3, 6, 10, 11, 13, 16, 17 
@ 
Other than Education Degree (3) 2, 8, 15 
Associates Degree (1) 10 
Bachelor’s Degree (4) 2, 3, 8, 13 
Master’s Degree (/1) 1,5, 6, 7,9, 11, 12, 14, 15, 16, 17 
Doctorate Degree (J) 1 
Civilian Service Computation Date (by years of experience) 
1984/ 36 years exp (1) 11 
1999/ 21 years exp (1) 10 
2004, 2006, 2007, 2008, 2009/ 11-16 years exp (8) | 2,4, 5,7, 9, 12, 13, 14 
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2010, 2011, 2015, 2016, 2018, 2019/ 1-10 years exp | 1,3, 6, 8, 15, 16, 17 
7) 


RQ1: Educational Background in Instructional Design 

Research question number one asked, “What type of education and/or training have 
instructional systems specialists (ISS) practitioners received as instructional designers?” The 
intent of the question was to gain a greater understanding of the diversity in academic and 
professional development backgrounds of those filling the positions of instructional systems 
specialist (ISS). To address this question, I reviewed the educational academic program and level 
of degree achieved. I also examined the type of professional development opportunities pursued. 

To address the educational academic program and level of degree achieved, I considered 
ID programs and adult education programs in comparison to other educational type programs or 
degrees achieved in programs other than education. As students within military training 
organizations must be 18 years of age to volunteer for service and are therefore legally 
considered adults, I chose to specifically review ID and Adult Education academic backgrounds. 
I also listed educational background experience outside of those programs as 
general/pedagogical or other. As previously mentioned within the review of the literature on ID 
practices, the ID profession has various titles (Klein & Kelly, 2018; Larson & Lockee, 2004; 
Reiser, 2001). Therefore, the programs identified as Educational Technology, Instructional 
System Design, or Instructional Systems Technology were all considered ID programs. The 


results of the educational background were as follows: 
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Figure 3 


Educational Background: Instructional Design 
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Of the 17 ISSs interviewed, five indicated an educational background specific to 
instructional design. One participant had completed a Doctorate in Educational Technology; 
three had achieved a master’s degree in Educational Technology; and one had completed a 
Master’s degree in Instructional Design. 

It was 2003. That’s when I decided to get my PhD in Ed Tech. Because what I found at 
that time as in my consulting practice, the web was really changing things in terms of 
employee incentives, in terms of employee engagement. My focus really turned into how 
do we design systems and performance systems to engage employees to improve lifelong 
learning, continuous learning, said one interviewee on the draw toward the ID profession. 
Two ISSs discussed the desire to seek an ID degree once they were settled in the position 

and discovered the complexity of the work. When asked about the path through their educational 
careers, Interviewee number four stated, “I got my master’s in Ed Tech, an education degree with 
an emphasis on technology because I wanted to say ‘well, I got this bachelor’s degree in 
computer science. I want that to count for something.’” Interviewee #1 commented, “At the time 


I had a Bachelor’s and Master’s in Engineering Management. I had done my Bachelor’s in 
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Electrical Engineering...after 1981 I switched from hardware, I went back and got an 
undergraduate and a Master’s degree in Computer Science.” 
Other interviewees reported similar diverse avenues to the ID profession: 


e Interviewee #2: “Bachelor’s degree in Business Admin.” 


e Interviewee #6: “Bachelor’s in Aeronautics with a Minor in Science.” 


e Interviewee #9: “Got my bachelors in organizational leadership then I got my MS 


from Troy University, right here, in adult education. I also have military background 
in military intelligence, foreign language, which is also conceptual.” 

e Interviewee #10: “Bachelor’s in Electronic Engineering Technology from the 
University of Southern Mississippi. [Also] the Air Force has an accredited 
community college program there so I got two of those. One is the instructor of 
military science, just an associate’s. The other one was electronic systems, I think, so 
that was two additional supporting [programs] with the electronics and technology 
and systems.” 


e Interviewee #12: “Bachelor’s in Computer Science.” 


Two of the ISSs indicated participating in professional development by enrolling in an 
instructional design academic program. Interviewee #2 stated, “I just recently enrolled to get my 
Master’s in Curriculum Design which I thought would help me with my position now as the 
developer instructor.” Another reported, “I am still considering getting another degree in 
educational psychology because it is extremely relevant to what we do especially if I come back 
to the individual training side.” [Interviewee #9]. This report may suggest that the ISSs 
recognized the complexity of ID tasks and competencies and wanted to feel more comfortable in 


performing them with a degree specific to the profession. 
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There may be several implications for the notion that the avenue through which persons 
come to ID can impact the approaches to and the perceptions of the work. For example, it can be 
inferred that engineering, computer science, and business administration hold more black and 
white, concrete perspectives and that education and organizational leadership hold more gray, 
abstract ones; that engineering is a hard science while education is a soft science. 

Two of the ISSs who were interviewed indicated an educational background specific to 
adult education both at the master’s level. One interviewee explained the draw to Adult 
Education: 

A Master’s in Adult Ed. I guess it’s the psychology behind it. How adults and the 

differences between, you know, how children learn and how adults learn and different 

methods of instruction. The significance of making sure that you know just all the 
different theories. Just being able to immediately apply what they [the students] learn, 
and correlate. Oh, and how emotions play a part in it. (Interviewee #9, personal 

communication, 11 April 2019). 

Interviewee number 14 reiterated the value of the psychology embedded in Adult Education. 
“So, it started as...it wasn’t an adult education course, it is an educational psychology class...so 
there was a lot more relationship focus and that sort of stuff, so I really enjoyed that class.” 

Seven ISSs indicated having completed full degrees in education oriented programs other 
than instructional design or adult education: one (1) associate’s degree, two (2) bachelor’s 
degrees, and four (4) master’s degrees. The list of other educational oriented degrees achieved 
included: Elementary Education, Secondary Education, General Education, Post-secondary 
Education, and E-learning and Higher Education. When asked to describe their educational 


background, Interviewee #3 commented, “I went to Troy in Dothan for my secondary education 
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degree. With that I took courses about assessment, reading, management, which didn’t help 
anybody on the planet because it’s something you have to be involved in.” Interviewee #2, 
“(courses in curriculum development] kind of geared toward k-12, but I’m trying to equate the 
adult side to it for research and stuff purposes. They’re all pedagogy.” While another said, “I’ve 
never gone through a teacher education curriculum. I’ve learned to teach through assimilation” 
[Interviewee #1], suggesting that there may not be as solid of a linkage between general 
pedagogically focused education programs and Instructional Design as currently assumed for 
hiring purposes. 

Three participants reported having completed training in programs other than education 
and qualifying for the ISS position once they had completed the requisite 24 credit academic 
hours in Education. One participant completed a piloted certificate program that professed to 
provide 26 credit hours in Instructional System Technology by providing classes on Instructional 
Design subjects every two weeks over a period of three months. The program was discontinued 
after the conclusion of the first pilot cohort. 

From the review of literature on ID practice and perceptions, I found that researchers 
have asked the question whether or not academic programs for instructional design prepare 
instructional design students to practice the art of instructional design in the workforce 
(Christensen & Osguthorpe, 2004; Cox & Osguthorpe, 2003; Dennen & Spector, 2007; Gray et 
al., 2015; Larson & Lockee, 2009; Osguthorpe & Osguthorpe, 2007; Parrish, 2009; Rowland, 
1993; Tracey & Boling, 2014; Williams et al., 2011). The data from this study substantiated the 
ID literature in that the ID academic programs did not seem to prepare professionals for the 


military training development workplace. In almost all cases, when asked, interviewees could not 
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recall details (e.g., courses or activities) about their educational programs; rather, they referred to 
learning how to performing ID task on the job: 


e Interviewee #13: “I took courses in how students learn, what is that? Pedagogy 


classes? I don’t know, man. [I took] all the classes you do. All the ones that you have 
to take, those 24 that we have to have. I had them. I definitely had them. I can’t 
remember all those classes.” 

e Interviewee #6: “Master’s in Education. Most of the classes that I took were mostly 
qualitative, quantitative research. I can’t remember all of those classes. I just 
remember a lot of writing.” This interviewee went on to state, “My degree was in 
education and at the time a lot of the classes I took, still some of them related to a lot 
of stuff that they requested but to absorb it takes a while. It takes about, it took me 
about two years because you have to go through each phase in order to actually 
understand it. I can understand the book knowledge of it but to truly understand it you 
have to actually do it.” 

All degrees were achieved between the years of 1993 and 2016. This may imply the potential for 
emphasis on varying educational techniques and approaches that reflect the period in which the 
academic pursuit occurred given that academic programs are influenced by advancing 


technologies, instructional strategies, and theories. 
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Interviewees were also asked what professional development opportunities they 
participated in to stay current within the profession. In general, only about half of the ISSs 
interviewed reported participating in professional development opportunities. Those 
opportunities, however, tended to be those that were convenient to scheduling, locally provided 
by the organization, and free to attend (i.e., Staff and Faculty courses, CP32 organized 
workshops or seminars, or the TRADOC CP32 East Carolina University certificate program in 
Instructional Systems Technology online). When asked about professional development 


participation, several interviewees commented as follows: 


e Interviewee #6: “I don’t, other than just stuff that I take with the CP32 workshops and 
whatever courses I feel would enhance my job performance. Other than that I 
haven’t.” 

e Interviewee #10: “Any workshops that come along through [CP32] of course...during 
the last two or three years, I’ve taken all those. I don’t think anything more other than 


what I mentioned.” 
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e Interviewee #13: “If I have to pay for it...I usually don’t attend too many courses that 


I have to pay for. Usually in the Army, [if they] pay for something for me, I'll go. I 
get all the free stuff first.” 

e Interviewee #15: “Other than that, not really doing anything official. I mean, I read 

and I talk to people who I...my wife was a high-school teacher for years and we still 
have people that we talk to who are active teachers. There’s college professors that I 
converse with on stuff when I meet them. It’s all informal, there’s nothing really 
beyond the workshops that I do. 

The reasons that these ISSs sought out immediate development events were varied. “It’s 
your ongoing professional development vs. just me sitting in my cubicle just working on the 
world according to me and what I can do” [Interviewee #1]. According to one interviewee, 
professional development was important for these specific reasons: 

I think the most challenging thing is to make sure that you are on top of your game. I’ve 

got a daughter in college. She shares the things that go on in her classroom and some of 

them, even at this pristine university, they’ ve got professors that are like halfway doing 
your job. In the military, when you’re dealing with military folk, especially those at the 
level that I instruct...most of them have a pretty broad experience of autonomy in what 

they do. You’re not gonna get away with that. So the challenge is to always make sure 

that I’m on top of my game. That my skill set is sharp. Not cutting, just sharp. 

[Interviewee #12]. 

Another interviewee acknowledged work obligations interfering with the ability to attend 
professional development opportunities because the daily work requirements get so busy it’s 


almost impossible to schedule. 
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Interviewees initially mentioned attending Staff and Faculty (S&F) courses provided by 
the Educational and Technologies Branch within the Directorate of Training and Doctrine 
(DOTD) in order to become familiar with the profession within this specific work environment. 
Staff and Faculty courses fell into two overarching categories: 1) instructor training; and 2) 
developer training. Instructor training was the only mandatory course and was required for all 
who are assigned as academic classroom instructors. Instructor pilots were not required to take 
the course. The developer courses are two-week (or less) courses that provide an introduction to 
curriculum development. While Staff and Faculty courses provided the foundational knowledge, 
most ISS placed their professional development opportunities in one basket: CP-32 Workshops. 
These workshops were coordinated by the program manager and centered on instruction or 
instructional design topics either observed as a need or indicated as a need by CP-32 personnel. 
Fourteen of the ISSs who were interviewed indicated participating in CP-32 workshop as time or 
topic permitted. 

One additional professional development trend that surfaced was participation in the 
Training and Doctrine Command (TRADOC) Instructional System Specialist Intern Program. 
Four of the ISSs interviewed for this study indicated that they had participated in the ISS Intern 
Program. The intern program hires personnel with the intentions of becoming an ISS at the 
conclusion of the two-year program. Personnel are hired at a GS-7 level, provided training and 
education acquiring a master’s degree (typically in adult education from Troy University if not 
already acquired). Interviewee #15 stated: 

Stumbled around and finally found an intern program for 1750 on USAjobs and applied 

in 2010. They just use your intern status for all those classes they have you go to and they 


approach you with, “would you help us teach [a staff and faculty course] during the 
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summer crunch that we have around here?” I was like, “not what I signed up for but 

whatever you need me to do.” 

At the end of the two-year program, personnel were placed in an ISS position as a GS-11, 
according to the needs of the Army. Four of the ISS who were interviewed stated that they had 
participated with the intern program. Three acquired a Master’s in Adult Education while in the 
intern program. One already held a Master’s in Education. This professional development trend, 
however, differed from the aforementioned professional development opportunities in that it was 
really more of a career development effort in order to gain employment as in Instructional 
Systems Specialist in the government sector. 

RQ2?: Instructional Design Competencies Performed 

Research question number two asked, “What International Board of Standards for 
Training, Performance, and Instruction (IBSTPI) Instructional Design (ID) tasks are performed 
more regularly by Instructional Systems Specialists (ISS)?” The intent of the question was to 
gain a greater understanding of the tasks and skills that were routinely performed and utilized by 
instructional systems specialists. Furthermore, the intent was to provide insight into the 
perceptions of the ISS performing the tasks. 

To address this question, I reviewed the IBSTPI ID Competencies. There were 22 
IBSTPI ID competencies broken down within five overarching categories: 1) Professional 
Foundations; 2) Planning and Analysis; 3) Design and Development; 4) Evaluation and 
Implementation; and 5) Management. To identify tasks and skills (1.e., competencies) performed 
or utilized by ISSs within military training organizations, I developed the Jnterviewer Worksheet 


For Perceptions and Practices of Army Training Instructional Designers (see Appendix D). 
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During the review of interview transcripts, I noted that not all IBSTPI competencies were 
performed by ISS either in part or in full and, therefore, I modified the Interviewer Worksheet to 
allow me to annotate full or partial use. Full performance is defined as performance of all 
indicators of the competency as listed. Partial performance is defined as performance of some, 
but not all indicators of the competency as listed. For example, if an ISS suggested that they 
performed research routinely but did not indicate the application of the research to the discipline 
of instructional design, it would be considered as partial performance. The review of the 
interview transcripts also provided insight into the perceptions of the competency participation 
and performance as the interviewee fulfills their role as ISS. The following is the resultant 
competency identification by IBSTPI competency by category. Each category will present, first, 
the findings of competency performance followed by professional perceptions of that 
performance. 

Professional Foundations Competencies 

While the IBSTPI ID competency categories encompassed the five phases of the 
Analysis, Design, Development, Implementation, and Evaluation (ADDIE) process, the board 
also included foundational and managerial competencies. The professional foundations category 
identifies five competencies upon which the successful execution of subsequent processes was 
based. They were, additionally, labeled to require either Essential or Advanced levels of 
expertise. Essential is defined, by the IBSTPI, as competent performance required by all IDers. 
Advanced expertise was defined as competent performance required by all experienced 


instructional designers. 
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Practice 

The five competencies that fall under the professional foundations category include: 
visual, oral, and written communication; application of research and theory; professional 
development; application of analysis skills and data; and response to ethical, legal, and political 
implications. The review of interview transcripts suggested that the identification and response 
to ethical, legal, and political implications of design in the workplace (IBSTPI, 2020) 
competency was not performed by instructional systems specialists in military training 
organizations. However, ISSs reported that 58% (or 10 out of 17) of ISSs routinely performed 
two competencies fully. They communicate effectively in visual, oral, and written form, as well 
as update and improve knowledge, skills, and attitudes pertaining to the instructional design 
process and related fields (IBSTPI, 2020). Full performance indicated that 10 ISSs mentioned all 
three forms of communication modes while one mentioned only written communication as it 
related to verb selection for learning objective writing. Ten ISSs also reported participating in 
some form of professional development, as previously mentioned. 

Also of note, seven (or 41%) ISSs referenced routinely conducting research. Research in 
this case was not specific to ID theories or strategies in relation to specific course design. 
Research in this case was specifically of Army circular, pamphlets, manuals, and/or regulations 
often in response to a request for information made by leaders or non-ISS personnel conducting 
training development duties. The task, research, as performed by ISSs is therefore considered to 
be partial task performance. 


e Interviewee #7: “Within the ISS there’s a small set up of regulations in which, when 


you’re researching for an answer, as an ISS you know pretty much the specific three 


to five regulations that you can go to find your answer. And usually they cross 
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reference each other pretty well. But then there’s also another three to five regulations 
outside of your common go-to regulations that may be applicable too. You should be 
able to go to that regulation the first time out and be able to find what it is you’re 
looking for.” 


e Interviewee #10: “I had to research where the current materials are...that the SMEs 


and these instructors have that I can use to translate whatever they have over into a 
cogent lesson plan that will be acceptable to Army U and the DOTD because they’ re 
more critical than these guys might seem. Locating the raw materials, which for me 
would be the PPT presentations, written sheets they might have laying on their desk, 
any files that might be on their desktop. All this stuff they might use that not in a 
central location.” 

e Interviewee #12: “The analysis phase was not that difficult because I was teaching the 
class that I was writing the lesson plan on a task that is very common. So basically 
what I had to do was research just to make sure that I had my research down. I went 
straight to design trying to put it together.” 

Interview respondents stated that research was conducted in response to a question posed 
by the Subject Matter Expert (SME) with whom they were working. Information gathered from 
research was shared, but no discussion of application to the discipline of instructional design was 
considered. Any research and theory conducted specifically within the field of ID was couched 
within discussions relative to professional development and therefore captured within that 


competency. 


Table 7 


IBSTPI Competency Performance: Professional Foundations 


PROFESSIONAL FOUNDATIONS 


Full 


Partial 


Comments 


1. Communicate effectively in visual, 
oral, and written form. 


10 


1 


2. Apply research and theory to the 
discipline of instructional design. 


1 


oe 


3. Update and improve knowledge, skills, 
and attitudes pertaining to the 
instructional design process and related 
fields. 


10 


Professional Development 
Discussed 


4. Apply data collection and analysis 
skills in instructional design projects. 


5. Identify and respond to ethical, legal, 
and political implications of design in the 
workplace. 


Competency not discussed 
within interviews by 
interviewees. Not mentioned by 
interviewer. 


Perception 
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By and large, the most directly applied competency is communicate effectively in visual, 


oral, and written form (IBSTPI, 2020). When asked what important skills were regularly used by 


ISSs, many replied that communication was paramount: 


e Interviewee #2: “Good communication skills, questioning skills, Teamwork. I get 


along, people skills are good. Communication, writing, I have to write the lesson 


plans or make changes. I mean to be an effective instructor, or teacher, you have to be 


able to talk to people, communicate your intent to them so that they understand. So I 


guess two way communication skills, questioning skills.” 


e Interviewee #3: “Writing definitely. You need to have very strong writing skills. I 


think that’s one of the biggest things, to be honest. You have to like it and you should 


probably be pretty good with that.” 


e Interviewee #4 “Task writing and breaking things up to be measurable, observable, 


achievable.” 
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e Interviewee #14: “I think you have to be a good communicator. Both 


written...probably more strongly on the written side, especially when we start getting 
into verbs and all that. Some people just don’t get it and I think that that is a skill. 
And then obviously you have to be able to verbally communicate because you have to 
work as a team, so you have to pass things on to other team members.” 

In terms of research, many ISSs reported conducting research but in a limited fashion, as 
noted above. One interviewee implied that a great deal of time is spend in the act of research, 
“Yeah, I would say about eighty percent of it is doing the research and reading [Regulation and 
Army Pamphlets] which you know is part of the research and then maybe twenty percent of it is 
the actual design part of it” [Interviewee #9]. Another interviewee simply noted, “I started at the 
design. I took all the research that they gave us” [Interviewee #4]. Another noted, “I will 
research within regulations if something comes up, somebody has a question....lately ve 
researched instructor-student ratios because there was a course that wanted to change the 
instructor student ratios” [Interviewee #7]. 

Other interviewees commented on the nature of research they conducted: 

e Interviewee #10: “I would have to do the research myself or they would tell me which 
book to go to because they’re busy. I had to get into the books and I had to locate 
what information is good, what’s relevant and what’s not relevant and then organize it 
and get it on a paper.” 


e Interviewee #15: “In terms of what I always do...you have this position description 


‘other duties as assigned’ and most of the day, typically, falls in that category. 
Responding to requests for information, discussing course changes and issues with 


other training developers and instructional system specialists and supervisor. I would 


63 


say over the course of a week, probably several hours of dealing with TDC to 

document whatever’s the result of all of those conversations. There’s probably several 

hours a week, I'd take three or four or five, typically is in research. 

Planning and Analysis Competencies 
Practice 
The planning and analysis category identified four competencies. The four competencies 

that fell under the planning and analysis category included: needs assessment, target population 
analysis, selecting analysis techniques, and analyzing emerging technologies. Interview 
responses suggested that analysis did not seem to be regularly conducted by ISSs. Analysis 
competencies were performed only 11% or less. Partial performance, in this instance, was 
suggested because interviewees recognized analysis as part of the ID process. Interviewees 
reported that analysis, however, was not performed in large part because curriculum design or 
training management decisions that impact curriculum design were directed by the leadership. 
Leadership decisions and directives were ultimately interpreted to mean that a needs assessment 
or target population analysis did not need to be conducted. Additionally, target audience 
characteristics were often assumed based on either assumed generational characteristics or 
personal experience with the job and tasks being analyzed performed (i.e., instructor pilot vs. 


maintenance test pilot). 


Table 8 


IBSTPI Competency Performance: Planning and Analysis 


existing and emerging technologies 
and their potential use. 


PLANNING AND ANALYSIS Full | Partial Comments 

6. Conduct a needs assessment in 1 1 Generally addressed as not 

order to recommend appropriate conducted as projects are 

design solutions and strategies. assigned by command 
direction or command 
guidance 

7. Identify and describe target 0 2 Discussed as a “should be 

population and environmental consider” for design 

characteristics. decisions but not normally 
performed due to lack of 
time. 

8. Select and use analysis techniques 0 0 Competency not discussed 

for determining instructional content. within interviews by 
interviewees. Not 
mentioned by interviewer. 

9. Analyze the characteristics of 0 2 Discussed as a “should be 


consider” for design 
decisions but not normally 
performed due to lack of 
time. 


Perception 
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More often than not, needs analyses were not conducted by ISSs. ISSs were responsive to 


requests for training development directed by leadership or instructor personnel. In many cases, 


the ISS were not privy to the needs, triggering the design or development request. Interviewee 


#14 commented, 


So the way that we look at our IMI products was the real analysis in the ADDIE process. 


For those products [analysis] is pretty much already done by the time it gets to us for 


development, for design and development. But we have our own analysis phase. It looks 


different than others, but it still needs to be done. You are not doing a job analysis, you 


are not doing needs or gaps. As that stuff has already been done and identified, and we 
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know what needs to be taught. And they give all of that to us. We still just evaluate it just 

to make sure it doesn’t need any tweaking. 

Interviewee number four reported that the requests for a change was enough of a need 
when they explained: “[I was given] the analysis information of what the need was. It wasn’t 
very thorough. It was the boss saying, “They need help’. There’s your needs analysis and all that 
stuff.” When asked to describe further the process of a needs analysis that was conducted, 
Interviewee #6 admitted: 

That was done at higher [headquarters]. So in essence it was done by [a key leader 

group]. I would say there was not thorough analysis done, because we were not included. 

So that decision was made higher. And so I didn’t realize higher had initiated until I 

actually started doing the [work]. 

These results were similar for target audience analyses. The review of interview transcripts 
suggested that most ISSs make target audience characteristic assumptions when they, 
themselves, had experience with the subject being worked. For example, Interviewee #1 shared, 
“Well, being that I’ve been an instructor since 2002, I know what the target audience is. I’ve 
taught many different courses”. When describing the target analysis procedures another 
interviewee stated: 

Basically we did a target analysis by deciding who our target audience was gonna be for 

the class. We had the Non-commissioned officer (NCO) academy and we’ve got Warrant 

Officer Basic Course (WOBC). So I mean we had a wide variety. We didn’t go full 
blown for our analysis. We just kind of said, “this is who our audience is gonna be, this is 
what you’re teaching, this is the range, the age range, and their rank range.’ We ran with 


it [Interviewee #2]. 
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In general, little if any analysis was conducted routinely or without prompting during any 
given project that was assigned for revision because it was either consider easy or unwarranted. 
Interviewee #15 thought: 

The most important [thing] that I see so sadly neglected in almost every case, is analysis. 

I’m pretty analytical. When somebody presents me with something, I say to myself or 

him, what does that mean? Or how do you mean that or where does that go? Where does 

that come from? Who told you that? Or just try to dig below the superficial level of 
what’s put in front of you and dig down two or three four more layers to find out what the 
real issue is, or what the fix is, or what’s going on...To me, the analytical ability and 
problem solving is the most important and most neglected piece. 

A second interviewee commented, 

The analysis phase was not that difficult because I was teaching the class that I was 

writing the lesson plan on a task that is very common. So basically what I had to do was 

research just to make sure that I had my research down. I went straight to design trying to 
put it together. After I got all my references, I knew what I wanna teach, who my 
audience was, and at what level you wanna teach, because of course the students are 
teaching this particular lesson. Then it was just basically going through the other four 

steps with the development, implementation, and evaluation[(Interviewee #12]. 

A third noted, “Well I think so yeah because they’ re so busy that they blow off the task analysis” 
[Interviewee #10]. 

The IBSTPI competencies under the Planning and Analysis category emphasize the needs 

and target audience analyses. This study’s interview respondents highlighted several other 


analytical forms that were also not performed fully. For example, when asked to expound upon 
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how specific steps were performed when describing the work on an assigned ID project one 
interview stated, “the [content] analysis was all cut and paste out of the [Army] Pamphlets 
(Pams).” Some ISSs reported receiving analysis data that were conducted by others: 
The [unit] guys are really good about doing the front-end analysis. It’s supposed to be us 
but things are flipped because of [a DA initiative] right now so they’ll send something 
and say hey I need concurrence with this or argue with me or whatever [Interviewee #9]. 
Still others suggested that test item analysis is conducted by non-ISS personnel, 
Then we’re going to do test analysis...It’s very time-consuming for [1712 series, training 
specialists] to sit down and actually correct a test. The test analysis is going to be high 
missed questions and near high missed questions. That’s pretty bad and the[1712 series, 
training specialists] are going to have to revise that [Interviewee #10]. 
One ISS reported to not enjoy conducting analysis: 
The only thing more boring than the A in ADDIE for me is watching paint dry. It is 
absolutely horribly boring. I totally hate my job right now. I’m doing job analysis. So I 
run the CTSSBs, that only happens once every three years [Interviewee #4]. 
This statement suggested that job dissatisfaction might have an impact on how an ISS 
approaches analysis. There may be many factors that influence this dislike for the task that can 
only be speculated during this study. Further research may be warranted. 
Design and Development Competencies 
Practice 
The design and development category identified seven competencies. The seven 
competencies that fell under this category included: utilization of an appropriate design process; 


organization of instructional programs and products; design of instructional and non-instructional 
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interventions; selection of instructional materials; and design assessments. Four of the seven 
competencies within this category were not performed by ISSs within military training 
organizations. These four competencies included: 1) use an instructional design and development 
process appropriate for a given project; 2) plan non-instructional interventions; 3) develop 
instructional materials; and 4) design learning assessments. ISSs within Army training 
organizations were prescribed the use of the ADDIE design process through the Training and 
Doctrine Command (TRADOC) Regulation 350-70, Army Learning Policy and Systems. 
Therefore, the ability to use an ID process appropriate for a given project was considered against 
regulation and not supported. While the ADDIE process was the instructional design process of 
record, experienced ISS personnel acknowledge that the Rapid Prototyping design process was 
actually practiced. The rapid prototyping design process is described as a process that quickly 
drafts design ideas and materials immediately followed by an implementation of the products to 
determine validity and reliability. Gaps in product performance were noted, modified, and 
continuously re-tried. The rapid prototyping design process was unintentionally employed, 
because of the resource constraints usually present within command directed or cyclically timed 
projects. Those resource constraints generally being that of time, equipment, and personnel skill 
sets. 

Seven (or 41%) of ISSs stated that they organized instructional programs and/or products 
to be designed, developed, and evaluated when organizing was interpreted as storyboarding 
curriculum design. Storyboarding is defined as a visual organizer of project content, sequence, 
and actions that provides a big-picture view of a design project. Four (or 24%) of ISSs reported 
experience with storyboarding either using PowerPoint for Visio programs, while three only 


mentioned storyboarding as an organizational tool to be considered in the design process but 
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admitted to not having produced a storyboard themselves. Two out of seven (or 12%) reported 
designing instructional interventions and selecting or modifying existing instructional materials. 
Those 12% were ISSs within the Staff and Faculty section of the Educational and Technologies 
Branch within the Directorate of Training and Doctrine. Staff and Faculty personnel periodically 
received updated instructor course materials produced by TRADOC or Army University 
(ARmyU) level entities and were directed to make modifications to the course activities and 
content to better address instructional and ID principles. 


Table 9 


IBSTPI Competency Performance: Design and Development 


DESIGN AND DEVELOPMENT Full | Partial COMMENTS 
10. Use an instructional design and 0 0 ADDIE is the TRADOC 
development process appropriate for a prescribed design process. 
given project. Rapid Prototyping seems to 


be the design process used in 
practice however. 


11. Organize instructional programs 4 SS) Interpreted as the use of 

and/or products to be designed, Storyboarding. 

developed, and evaluated. 

12. Design instructional interventions. 2 0 

13. Plan non-instructional 0 0 

interventions. 

14. Select or modify existing 2 0 Primarily within Staff and 

instructional materials. Faculty section with the 
modification of instructor 
courses/materials. 

15. Develop instructional materials. 0 0 

16. Design learning assessment. 0 0 


Perception 

Upon review of the transcripts, many of the 17 ISSs who were interviewed referenced 
participating in the design and development phases of the ADDIE process. When asked to 
describe how they progressed through the process on an ID project on which they worked as an 


ISS many comments were reminiscent of the ID practices literature’s reference to the differing 
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approaches to design between novice and expert performers. ISSs described more expert level 


behaviors stating they started with solutions rather than spending time defining the problem 


(Tracey & Hutchinson, 2013). 


Interviewee #1: “I have a sketch in my head for each lesson level. In an excel, I 


mapped out all the TLOs and ELOs and I had to make sure there was alignment 
between the objectives themselves.” When asked to describe the process further this 
interviewee stated, “It was taking the existing materials, mapping the objective, 
correcting the objectives, rewriting the objectives to make sure that they were in 
alignment with the proposer competency or mastery levels for the learning domain, 
validating content again those objectives, and then rewriting all the lessons for better 
alignment” 

Interviewee #4: “Intuitively, I started at the design. I took all the research that they 
gave us...I really took advantage of having a creative control for these workshops. 
It’s fully engaged. I’m mixing it up. We’re doing gallery walks and were doing all 
this, and I even had a speed dating little [activity], video, and we had case studies. I 
was really trying to mix it up. I wanted to diversify the methods of instruction so i 
made sure that none of my methods of instruction were the same.” 


Interviewee #7: “I would say we’re somewhere between the analysis and design. 


We’re doing an analysis as to where it [content] should be taught, who should get it, 
how much information needs to be taught to that individual and then the design phase 


is what the course needs to target that, specific individuals.” 
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The interview respondent comments also suggested that in some cases even the design 
tasks are skipped and the process begins with development. Consider the following comment 
from Interviewee #3: 

[Supervisors are] like, “no. Get the job done. A, B, C, done. And then call it a day.” 

They’re just all about efficiency. And we’re more of the...it’s almost like an artistic form. 

We come in and we say, “okay, well, that’s great but have you thought about maybe 

changing it up a little bit more.” 

In addition, the following quote from Interviewee 12 communicated how they initiate the work 
within the development phase: 

[In the] development phase, okay. So, now let’s start typing this thing out and see where 

this is all gonna go. You know how we do that, the backward planning thing. When I start 

building, I’m gonna start with my questions and work my way back. 
Evaluation and Implementation Competencies 
Practice 

The evaluation and implementation category identified three competencies. The three 
competencies that fell under this category were: evaluation of instructional and non-instructional 
interventions, revision of the same based on data, and diffusion of instructional and non- 
instructional interventions. Of the three competencies, the ISSs who were interviewed only 
reported performing one of the competencies either in full or in part. They reported that while 
they and other ISSs routinely reviewed the work of others, they rarely made the revisions to the 
products themselves. Instead, ISSs stated that following a review they would relay recommended 


changes or improvements to the individual or individuals who had developed the product. 
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The ISSs also reported that those individuals to whom they relayed the information were 
typically the instructor, the instructor writers, or the 1712 series personnel. That said, 16 of the 
17 (or 95%) ISSs who were interviewed stated that they “evaluated instructional interventions” 
(IBSTPI, 2020). Non-instructional interventions, however, were not evaluated simply due to the 
fact that non-instructional interventions were generally not considered, planned, or produced by 
CP-32 personnel. Evaluation, in this scope of this study, represented an ISS’s (1750 series) 
review of the curriculum materials (i.e., PowerPoint, lesson plans, student handouts, and 
assessments) produced by instructor or instructor writers (1712) for adherence to TRADOC 
accreditation standards or their review of work produced as a requirement to fulfill a Staff and 
Faculty course activity. 


Table 10 


IBSTPI Competency Performance: Evaluation and Implementation 


EVALUATION and 
IMPLEMENTATION Full | Partial Comments 

17. Evaluate instructional and non- 4 12 Interpreted as “the review of 

instructional interventions. the work of others (1712s)” 
or the mention of validating 
designed materials. 
Validating is not conducted 
as “textbook” or was either 
discussed as how it should 
be executed. 

18. Revise instructional and non- 0 0 

instructional solutions based on data. 

19. Implement, disseminate, and 0 0 

diffuse instructional and non- 

instructional interventions. 


Perception 
As discussed in Chapter Two, series 1712 positions did not possess the same educational 


background requirement in order to fill the position. The majority of ID related work performed 
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by ISSs was that of reviewing the developed course materials produced by those in instructor, 
instructor/writer (job series 1712), or contractor positions. The following interviewees reported: 


e Interviewee #2: “So [I] just kind of read through and see what they have [developed], 


look through the references. We wordsmith it just to get it to flow right,” harkening 
back the Professional Foundations category written communication competency. 

e Interviewee #3: “There’s some rewording but we have to be very careful in how we 
reword things because there’s a certain level of comfort and there can be some 
monotony with different verbs, I’ve noticed.” 


e Interviewee #7: “Okay, so yesterday, a typical day yesterday, I was reviewing lesson 


plans. The training developer [1712] that was down there had brought the six lesson 
plans, which he said that they were completed. So my day yesterday, we spent 
reviewing lesson plans. I take the PDF and I would make my comments within the 
PDF back to the individual and say this is my suggestion, my idea, my corrections, 
the training developer would then go in TDC and he would make those adjustments 
or he may not make those adjustments and may come back with an email to me 
asking questions.” 

e Interviewee #12: “Now, for someone who has a degree, whether it’s in adult 
education or instructional design, you’ve learned about these [theorists] but you’ve 
also learned that there’s a whole slew of other folks out there and there’s a number of 
other schools of thought that address the same issues. They may have that broader 
understanding but when you put 16 [military] students in a classroom...and you’re 
teaching them ELM...that’s all they hear and that’s all they know. That’s the way we 


teach. And so, you kind of can get blinders. It’s a two edged sword, because the Army 
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can’t paint with that broad brush. But maybe there needs to be an understanding that 
this is not the only way and this is not the only approach.” 


e Interviewee #13: “So we do the [Critical Task Site Selection Board] CTSSB, we get 


the results, we get the tasks, we vote on it and make sure these are the tasks we want 
to work on. Then we do contractor. We send it out for bid, the contracts are awarded, 
we meet with the contractor to kick off meetings and then we go into sitting down and 
discussing what we want to do to how to build a course.” 

e Interviewee #14: “So [for this project] I was given the lesson plan and basically 
reviewed the lesson to make sure that it was the structure that we needed to stick 
with, the material flowed well, and the topics meshed well and would be able to flow 
within the IMI product that was created.” 

As with the responses from participants who were asked about tasks performed, these 
interview responses also seemed to suggest a mismanagement of personnel professional skill sets 
in that job competencies were not fully engaged or leveraged for the organizational or 
educational advantage. 

Management Competencies 
Practice 

The management category also identified three competencies: application of business 
skills, managing partnerships, and managing instructional design projects. These were the only 
competencies on the IBSTPI ID competencies list that required an expert level for managerial. A 
managerial expertise level was defined by the board as a required competency for all IDers with 
supervisory experience or position. Nine of 17 (or 53%) of ISSs interviewed stated that 


“managing partnerships and collaborative relationships” (IBSTPI, 2020) was a routine effort. 
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ISSs scheduled face or social collaboration time to visit the SMEs and the units with which they 
worked. They reported working to establish a positive and helpful persona as well. Of the ISSs 
interviewed, 80% reported using business skills to manage instruction design functions. Business 
skills were described, however, by ISS as feeling more clerical in nature. They reported that a 
primary role of an ISS was as a data entry clerk within the training development repository of 
record which was the Training Development Capabilities (TDC) database at the time of this 
study. 


Table 11 


IBSTPI Competency Performance: Management 


MANAGEMENT Full | Partial Comments 
20. Apply business skills to managing 2 1 
the instructional design function. 
21. Manage partnerships and 9 0 
collaborative relationships. 
22. Plan and manage instructional 0 1 
design projects. 


Perception 

When asked to articulate the most important ID skills utilized by an ISS on a regular 
basis, ISSs reported a myriad of skills but building strong working relationships, along with 
communication skills from the Professional Foundations category, was the most recurring skill. 


The following interviewees stated: 


e Interviewee #2: “I feel that people skills, if you’re nice to somebody, you’re 
respectful to them, the way you talk to them, the way you treat them you get more out 


of them than just being up on teaching them.” 
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Interviewee #6: “So if you’re going to go into an organization and change the way 


they operate and do things...they put up their defenses and it’s hard to get buy in if 
you’re not one of them.” 

Interviewee #7: “Sometimes I go down to the courses and talk with the course 
managers about what’s happening within their courses. Even if I’m not doing 
anything with their course, I still like to go over to the course managers and touch 
base to let them know, I’m in the areas.” 


Interviewee #9: “You have to be able to collaborate. You have to be able to network.” 


Interviewee #10: “One thing I had is like networking. Sometimes I’m kind of a shy 
person with new people so it’s difficult for me to force myself to meet the new 
instructors and get to know them on a personal basis, so to form a good working 
relationship, to be able to communicative to get what I call, again, the raw materials 


that I need.” 


These comments seemed to imply that a certain measure of instructional design success was 


contingent upon the adoption of newly designed products or instructional recommendation into 


the normal routine of the training environment. Building positive relationships improved the 


likelihood of that occurring. 


In terms of personnel management, however, was the degree to which ISS perceived 


themselves as conducting administrative or data entry-level type work rather than performing the 


more complex competencies of ADDIE and IBSTPI. When asked to describe the events of a 


typical day, several of the interview respondents detailed the following events: 


Interviewee #4: “A day in the life is really just right now I’m not being used properly 


as an ISS. Put out fires, treated like a section secretary. I do a lot of admin work. I get 
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tasked with the spreadsheets. I didn’t get to engage in the learning because I’m just 
expecting to sit there and take all the information.” 


Interviewee #6: “Well, I had two ongoing projects, One...where we had a needs 


analysis that required that we take this one MOS and make it into two MOSs. And we 
produced course administrative data (CAD)s on what those courses would look like 
coordinated with Organization and Personnel Force Department (OPFD).” 
Interviewee #7: “I know I have to keep track of lesson plans in Training Development 
Capabilities (TDC), in the system [of record]. I start reviewing it within TDC, I will 
open up a lesson plan and outline and I will review it as a PDF document. And I will 
start, from the title, and the number, and the next thing on the same page is a foreign 
disclosure number. I just go through every section as it comes up. And I got to check 
the tasks, is the task taught? Is it approved?...then I’1l move down to the hours of the 
lesson, does it match the hours on the course map? Does it match the hours on the 
course schedule that the course manager has? Then I go down to the instructor action 
hours and then everything that come ups.” 


Interviewee #10: “Usually, I come in and check email, of course, respond to email if I 


need to, check the calendar and then organize the work for the day, what I have to do. 
Right now what I’ll have to do is usually see what the status is for, right now it’s 
lesson plan development.” 

Interviewee #12: “Coming back [after teaching] there’s several other duties that I do. 
I’ve got the Instructor of the Year program. I had the instructor training records, and 


I’m the DTMS manager DTMS is a computer database that you can load all of your 
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course information and iteration dates, and what else do I do? Oh yeah, I’m the 
Faculty Development Recognition Program (FDRP) coordinator.” 


e Interviewee #14: “[I’m a supervisor] I am not actually doing any of the work. I am 


just overseeing [subordinates] to make sure they are staying on track and setting up 
meetings, making sure if we are doing briefing slides, making sure that they are on 
track with those and the briefing meets the intent of what we are trying to discuss. 
And then, you know, of course, looking ahead at our own internal milestones 
schedule project plan. So that is pretty much a day in the life of a training 
development ISS.” 
When asked to share the most important skills that they used in their position as an ISS, one 
interviewee responded, “Computer Skills, Computer Literacy, Excel, Word, Microsoft Office 
basically” [(Interviewee #2]. 
RQ3: Instructional Design Competency Expertise 
Research question number three asked, “What is the perceived level of expertise required 
for performance of instructional design tasks by instructional systems specialists (ISS)?” The 
intent of this question was to gain greater insight into the experience required to successfully 
perform each task. Based on the interview responses, however, the data suggested that the level 
of expertise was not critical to ISSs performing instructional design tasks or competencies. ISS 
responses suggested that experience within the field, regardless of the organizational work 
structure, determined expertise. When asked to indicate to what level of expertise they thought 
identified ISS skills should be performed, one interviewee commented, 
I have never been involved in [developing] something from the ground floor up. I have 


always come in mid, whatever, and when you ask to see things they’re like well we don’t 
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have that, so this is what we did. Okay. So, you know, garbage in, garbage out 

[Interviewee #2]. 

ISSs also suggested that experience was dependent upon both the professional and the 
projects to which they were assigned to work. There was a suggested difference between 
educational and/or theoretical expertise and practice expertise: “I guess maybe it depends on 
where you’re working at in the military. To be honest, where I’m at right now you don’t have to 
be an expert at all” [Interviewee #4]. As interviewee seven mentioned, “I guess it would depend 
on the level at which the individual is positioned...because you got more exposure from different 
things.” This suggested that some ISS believed that it was possible to know instructional design 
theories and principles but not be able to apply them practically. 

In contrast to person and project-borne expertise, some ISSs considered time in position 
as a determinant to expertise. Time in position equating to expertise varied, however, between 
interview responses. The range in time to acquire expert level performance was anywhere 
between one to five years. 


e Interviewee #1: “I know I’m an expert. I’ve been designing for six years in formal 


education. I make Instructional Designers. I know my content, my material.” 

e Interviewee #3: “For me personally I think it would probably take me five years, 
despite my military experience, yeah, before I become where I’m comfortable. That’s 
just what I that I’m going to need to at least have five years to really be proficient and 
comfortable in my own proficiency.” 


e Interviewee #6: “This has been a learning curve. I mean, I had to pick this up. I didn’t 


have a degree in design. My degree was in education and at the time a lot of the 


classes I took, still some of them related to a lot of stuff that they requested but to 
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absorb it takes a while. It took me about two years because you have to go through 
each phase in order to actually understand it. 


e Interviewee #10: “It took me a long time to get used to being an ISS and especially 


going from Air Force culture to Army culture. It was a tough transition.” 
Summary 
In conclusion, the results of this research suggest that personnel in instructional design 
positions within military training environments come to the profession from various educational 
as well as government/military backgrounds. Few enter into the ISS position with specific ID 
education or experience. ISSs are learning how to do their jobs on the job and are often taught 
how to do their jobs from non-ID personnel. Additionally, the Army ISSs in general performed 
modified or re-interpreted definitions of only half of the 22 IBSTPI ID competencies. In contrast 
to the position descriptions and the educational requirement under which they are hired, ISSs are 
not performing ADDIE process related competencies with any sense of regularity. Finally, task 
expertise was suggested to be acquired with time in the profession, along with experience 


working design projects. 


81 


Chapter Five: Discussion 

This chapter discusses the analytical results and their implications, which were obtained 
from the in-person interviews that were presented in Chapter Four. This chapter briefly re- 
introduces the purpose and significance of the study, summarizes the literature review on ID 
practices, and offers a brief description of the research design and methodology. Finally, the 
implications of findings for each research question are discussed in depth, including possible 
implications that this research may have on the practice and perception of the profession. 

Study Purpose and Review 

The purpose of this study was to directly respond to a review of the literature on ID 
practices and perceptions as it pertained to the professional practice and perceptions of 
instructional design within government or military training organizations. It was intended to 
encourage wider educational and performance considerations when training new instructional 
designers for the workforce and inform effective personnel management decisions. The existing 
literature highlighted the view of the role of novice versus expert performers within the field of 
ID. It also highlighted the consideration of re-imagined academic programs that reflected more 
practical, real-world application versus theoretical prowess in instructional design principles and 
competencies (Christensen & Osguthorpe, 2004; Cox & Osguthorpe, 2003; Dennen & Spector, 
2007; Gray et al., 2015; Larson & Lockee, 2009; Osguthorpe & Osguthorpe, 2007; Parrish, 
2009; Rowland, 1993; Tracey & Boling, 2014; Williams et al., 2011). 

This study provided additional information within the interview transcripts to support the 
continuation of that professional conversation. Instructional Systems Specialists (ISSs) were 
performing ID work differently in the military training world that what they are taught. Within 


this study, this seemed to occur for two distinct reasons. First, the ISSs in the military or 
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government arena did not all have educational backgrounds in ID and general or pedagogical 
educational programs did not appear to prepare the individuals’ transition as seamlessly as 
assumed to the performance of ID work. Second, there was a general misconception of the role 
of the ISS in the workforce. 

This study highlighted the issue that in many cases, Instructional Designers (IDers) in the 
military or government sectors were not performing ID specific duties or competencies. Most of 
those duties and competencies appear to be performed by other individuals without specific 
training or experience in ID. In order for the results of this study to have any significant impact 
on the academic program restructuring conversation, it would be advantageous to first address 
job performance. It is difficult to contribute to the conversation of restructuring academic 
programs to reflect the “real world” job execution if ISSs are not performing job relevant tasks in 
their current positions. 

The results of this study suggested that ISSs reported participating in rather than 
performing only 14 out of 22 of the International Board of Standards for Training, Performance, 
and Instruction (IBSTPI) ID competencies. The results also suggested that the three 
competencies that were performed in full by over half of those interviewed were competencies in 
the Professional Foundation and Management categories and outside of the ADDIE process, the 
Army ID process of record. Those three competencies were communication, professional 
development, and relationship building. The research also suggested that personnel in ID 
positions within military training environments entered the profession from various educational 
and military backgrounds. Results suggested that task expertise was acquired with time in the 


profession along with experience completing design projects. 
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Many ISSs coming from diverse education backgrounds that also had military experience 
reported to have limited or no ID experience outside of the military/government sector. Coupled 
with the fact that ISSs were not being used to perform ID work, time in profession might indicate 
expertise with products, databases, Army regulations, or Army language and procedures as 
opposed to expertise specifically in ID. A combination of backgrounds outside of ID could have 
an impact on the individual’s approach to design of military training development projects. To 
better understand how an individual would fulfill a position and the manner in which they 
approach their work, it is important to be aware of their background. 

Educational Background in ID 

Research question number one investigated the academic subjects studied, the degree(s) 
achieved, and the pursuit of continued professional development. ISS personnel seemed to arrive 
to the profession via a general or pedagogical area of emphasis. These programs focused on 
delivery of instruction versus design. They also focused on child education versus adult 
education. Many were teachers in the elementary and secondary education settings prior to their 
career as ISSs at the United States Army Aviation Center of Excellence (USAACE). 

This background and approach could ultimately have an impact on how well an ISS 
transitioned into working as a designer for the military. Expectations of work may be skewed and 
may confound the execution of ID work being performed. Novices might learn how to conduct 
design work based on what a supervisor or co-worker (often lacking in ID experience) modeled 
for them rather than on instructional design principles. Thus, there may be a need for improved 
organizational understanding and expectation of the role of ISSs. It may also be important to 


consider establishing peer-to-peer mentorship relationships. 
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Another factor that may have influenced the ID approach was the educational subject 
matter studied prior to studying education. Many ISSs had backgrounds distinctly different from 
education such as engineering and computer science. Several interviewees purported to have 
stumbled into the profession or were already educators before they decided to pursue an 
advanced educational degree. 

Overall, the ISS population was well-educated with 12 out of 17 interview respondents 
having obtained a master’s degree or higher in an education-related field. They also tended to 
seek out continuous professional development opportunities, either academically or through 
locally programmed workshops and seminars. This finding suggests that the population was 
dedicated to and enjoyed their profession. The data also suggest that there was an increase in 
those seeking degrees in ID type programs. This suggested an insight, by the professionals, for 
the need for the knowledge, skills, and attitudes specific to ID. This insight, however, seemed to 
evolve as ISSs began working and were confronted with the growing emphasis on curriculum 
design in order to address training direction and efficiencies to meet mission needs such as large- 
scale combat operations. 

There are several implications from these findings. First, organizations could experience 
a dilution of professional skill sets that are needed. For example, the ISS position description 
reported one duty requirement as, “Analyzes assigned projects to determine appropriate 
methodology, required research, required subject matter expert assistance, and the need for 
educational surveys & related matters,” (FASCLASS, 2005) but only five out of 16 (or 29%) 
ISSs interviewed reported knowledge of needs assessment, target population, or emergent 


technology analyses. Only one ISS reported to have actually performed a needs assessment. 
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Establishing a profession that included individuals with various backgrounds and 
experience tended to de-emphasize jobs and duties that were performed with any targeted 
expertise, thus creating a generalized versus a specialized workforce. In contrast, position 
descriptions (along with competency lists), suggested the desire from the hiring organization for 
an individual to enter into the position well versed in all areas of ID: analysis, design, 
development, implementation, and evaluation. The results from this study suggest that more 
emphasis was placed on the fulfillment of general educational academic credits than the task 
performance capabilities outlined within the position description or competency list. 

A second implication involved the variety of job titles and responsibilities held by 
instructional systems specialists. Interview data suggested that ISSs were supervisors, course 
managers, instructors, or curriculum developers, all of whom were expected to fulfill completely 
different job roles and responsibilities. Having a diluted workforce made job and project 
assignment much more difficult. 

The Department of the Army may benefit from considering new educational 
requirements specific to instructional design and the development of a formal and robust peer-to- 
peer design mentorship program that pairs novice designers (with non-ID specific experience) 
with expert designers (with ID experience). The Army, Training and Doctrine Command 
(TRADOC) organizations, and professional organizations such as the International Board of 
Standards for Training, Performance, and Instruction IBSTPI) could benefit from considering 
distinct duties, competencies, and approaches between design and development processes and 
procedures. Participants in this study implied that design was indistinguishable from 


development, or that a clear understanding of how design is conducted was lacking. 
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Instructional Design Competencies Performed 

Results of this study suggest that military or government instructional designers are not 
performing the duties and competencies laid out within the IBSTPI competencies list or within 
the position descriptions. Results suggest that there were only three ID competencies within the 
IBSTPI list that were performed by ISSs with any regularity: 1) communication; 2) professional 
development; and 3) relationship building. These three competencies that were referenced by 
interviewees did not include the analysis, design, and development categories which are the 
majority of competencies that define ID work. 

Table 12 describes the comparison between the everyday practices of the ISSs (based on 
the interview results) and how those practices are related to both position description duties and 
IBSTPI competencies. Highlighted items in the table indicate those that are linked between each 
of the five IBSTPI categories. This comparison illustrates the lack of ADDIE specific tasks 
performed by ISSs that are included in the IBSTPI competency list as well as the ISS position 
description. 


Table 12 


IBSTPI, Position Description, and Practice Comparison 


IBSTPI Category ISS Position Duty Job Tasks 
Category Competency Description Performed 
1. Communicate e Conduct research 
Professional effectively in visual, of Army 
Foundations oral, and written form. publications in 
2. Apply research and response to 
theory to the discipline requests for 
of instructional design. information by 
3. Update and improve leaders and non- 
knowledge, skills, and ISS personnel 
attitudes pertaining to conducting 
the instructional design training 
process and related development 
fields. duties. 


4. Apply data collection 
and analysis skills in 
instructional design 
projects. 

5. Identify and respond 
to ethical, legal, and 
political implications of 


design in the workplace. 


e Reading updated 
Army 
Publications 

e Write memos, 
briefs, learning 
objectives that 
are 
grammatically 
correct. 

e Explain design 
process to non- 
design personnel 

e Attend local 
CP32 workshops 
and seminars 


Planning and 
Analysis 


6. Conduct a needs 
assessment in order to 
recommend appropriate 
design solutions and 
strategies. 

7. Identify and describe 
target population and 
environmental 
characteristics. 

8. Select and use 
analysis techniques for 
determining 
instructional content. 
9. Analyze the 
characteristics of 
existing and emerging 
technologies and their 
potential use. 


e Acts as independent 
action officer or 
project officer, & 
advisor. 

Analyzes assigned 
projects to determine 
appropriate 
methodology, 
required research, 
required subject 
matter expert 
assistance, and need 
for educational 
surveys & related 
matters. 

Serves as the principal 
point of contact for 
design actions within 
the scope of assigned 
responsibility. 
Applies established 
criteria in selecting 
tasks for training & in 
recommending 
appropriate 
instructional setting. 
Collects data 
necessary to support 
design projects. 
Reviews & analyzes 
internal and external 
feedback & applies 
results to determine 
adequacy of task 
analysis and training 


e Express 
knowledge of or 
explain the 
appropriate 
analyses and 
analytical 
procedures 
needed for ID 
projects 

e Evaluate 
analyses already 
conducted to 
determine if data 
needs 
modification 

e Cut and paste 
from Army 
publications for 
content analysis 

e Job/Task 
Analysis updates 
through Critical 
Task Site 
Selection Boards 
(CTSSB) 
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program 
documentation. 
Evaluates adequacy 
of material collected 
& initiates requests 
for additional data. 
Design and 10. Use an instructional Write regulations, e Storyboard course 
Development design and development pamphlets, procedure flow from 
process appropriate for a guides, policy existing course 
given project. documents, & SOPs. materials. 
11. Organize Develops, coordinates, | ¢ Write TLOs 
instructional programs recommends e Input course 
and/or products to be approval, & makes materials into 
designed, developed, changes to: Training 
and evaluated. o Programs of Development 
12. Design instructional Instruction (POD), Capabilities 
interventions. o Individual (TDC) Database 
13. Plan non- Training Plans 
instructional (ITPs), 
interventions. o Course 
14. Select or modify Administrative 
existing instructional Data (CADs), and 
materials. o Course 
15. Develop Management 
instructional materials. Plans (CMPs) for 
16. Design learning o Training course 
assessment. materials (i.e. 
presentations, 
lesson plans, 
student handouts, 
assessments) 
Evaluation and | 17. Evaluate Reviews, coordinates, e Teach Staff and 
Implementation | instructional and non- or initiates surveys & Faculty 
instructional reports to appraise courses/mentor 
interventions. impact on assigned non-ISS 
18. Revise instructional area of operations & personnel on 
and non-instructional programs. design process. 


solutions based on data. 
19. Implement, 
disseminate, and diffuse 
instructional and non- 
instructional 
interventions. 


Prepares 
correspondence in 
support of assigned 
projects. 

Evaluates & provides 
guidance & direction 
a course development 
proposals, training 
device requirements, 
new training media, 
course design, & 
training materials. 
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e Determines methods 
of solving training 
problems pertaining to 
the implementation of 
training concepts, 
techniques and 
procedures. 


Management 


20. Apply business skills 
to managing the 
instructional design 
function. 

21, Manage 
partnerships and 
collaborative 
relationships. 

22. Plan and manage 
instructional design 
projects. 


e Establishes & 
maintains historical 
audit trails for all 
actions. 

e Prepares & presents 
briefings. 

e Writes studies & 
correspondence as 
required. 

e Uses initiative and 
judgment in applying 
& adapting broad 
educational principles, 
general administrative 
policies, & limited 
guidelines to the 
development & control 
of training Programs 
of Instruction (POIs). 

e Manages training 
development actions 
with other activities, 
organizations, & 
agencies. 

e Develops & 
recommends 
command position on 
assigned subject & 
areas. 

e Evaluates impact of 
new software & 
equipment in advance 
of initial training, 
doctrine, evaluation 
results, & long range 
trends. 

e Recommends cost 
effective training 
strategies. 

© Coordinates work 
with higher 
headquarters, other 
education or training 


e Review training 
development 
products created 
by non-ISS 
personnel. 

e Recommend 
changes or 
improvements to 
materials 

e “Wordsmith” or 
“reword” 
reviewed 
products 

e Manage 
partnerships and 
collaborative 
relationships 

e Schedule 
meetings 

© Contribute to 
information 
briefs by 
leadership 

e Write 
memorandums 

e Input course 
materials into 
Training 
Development 
Capabilities 
(TDC) Database 

e Develop 
spreadsheets 
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specialists, 
counterpart action 
officers, instructors, 
Subject Matter 
Experts (SMEs), 
TRADOC schools, & 
other appropriate 
agencies to plan, 
develop, & 
recommend solutions 
to training problems. 

e Reviews, comments, 
& makes 
recommendations on 
education, training 
directives, & contracts 
for training 
development, 
evaluations, and 
reports. 


Note: Adapted from Kelly (2016) and FASCLASS (2005). 


There are several implications from this finding. First, ISSs (1750s) and 
Instructor/Instructor Writers (1712) appear to be performing the same job duties. ISSs were 
being used to review the design and development work produced by other non-design or 
contractor personnel rather than designing products themselves. Second, ISSs with ID specific 
education and experience seemed to exhibit higher levels of frustration and job dissatisfaction. 
Finally, ISSs appear to spend a great deal of time mentoring novice ISSs and non-ISS personnel 
on ID principles, theories, and application. 

ISS vs non-ISS work 

Interviewees commented on the matter of non-ID personnel (1712 series personnel or 
contractors) producing course materials with little interaction with the designer. Instead of 
designing training for students, ISSs reported more managerial, administrative, or data entry 
work. Interviewees reported spending a majority of time uploading information from training 


products developed by non-ISS personnel (1712 series or contractors) into the Training 
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Development Capabilities (TDC) database. The TDC database supports resource management 
decisions by the TRADOC organization. ISS Interviewees also reported a wide variety of non 
ID-related duties, including: 

e producing or reviewing memorandums for distribution. 

e managing contracts and reviewing training products produced by contract employees. 

e planning and scheduling meetings in which they often become the note takers. 

e keeping track of Program of Instruction (POI) instructor contact hours (ICH) and Instructor 

Action (IA) numbers. 
e entering those numbers into the worksheet that calculates manpower requirements for the 
personnel management office. 
ISS vs Job-Satisfaction 
In general, ISSs reported enjoying ID work, but they expressed frustration due to their 
perception that they were not allowed to perform duties that were related to ID. They 
acknowledged the intended role of the ISS and were able to describe the types of tasks they 
wanted to perform, but they did not believe they were being asked to perform those types of 
tasks. In fact, the belief of the majority of ISSs interviewed is that the position description is 
misleading, and that there is little alignment between the position description and their job duties. 
In addition, ISSs indicated that they do not feel challenged by the work they typically perform, 
and that the possibility of promotion within the profession is limited because they are not 
performing ADDIE process tasks. 
The overarching implication is that ISSs have been considered an afterthought within the 

ID process rather than active contributors or leaders in the process. They have been acting as 


quality control personnel with little influence in the training development process, and they have 
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had little influence on training development decisions that consider quality of training that match 
target audience, content, environment, media, and instructor capabilities to a desired outcome. 
ISSs have been acting in more of a summative evaluation role rather than a formative one. 
Mentoring Others 

In addition to identifying ISS performed competencies, there were also several 
competencies that emerged from the data that were not otherwise captured by the list of IBSTPI 
ID competencies. Those competencies included mentoring others in instructional design 
principles and practices, and teaching and classroom management. Novices need mentorship 
within the work environment from experienced practitioners and experiential learning 
opportunities in order to recognize the tools and techniques of applying theories and principles to 
real-world projects (Tracey & Boling, 2014). Thus, the Army may want to consider encouraging 
mentoring relationships for personnel without ID experience (i.e., military or those without 
educational backgrounds). ISS personnel could translate knowledge of theories and models 
to practice for those outside of the training development offices (Tracey & Hutchinson, 
2013). 

Recommendations for Future Research 

The findings in this study only suggest practices performed by and perceptions held by 
the instructional systems specialist within the field. They are not generalizable to the profession 
of ID as a whole. They do not suggest a value on performance or perception, particularly when 
considering the military versus educational background of those in the position. This study does, 
however, warrant additional research to gain further, more definitive insights into the knowledge 
claims made within this research. Future studies on the impact of educational or military 


background on design approach and product development may be extremely valuable for the 
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Army organization when considering hiring reforms, position duties and responsibilities, and 
project assignment for its personnel. 

Results of this study suggest the need for continued consideration of academic program 
structures that support practical application in the real world rather than making direct linkages 
to the conversations found in the literature on the practice of ID. This may be due in part to the 
fact that the ID practice literature acknowledges the need for the conversation but has not yet 
focused on the question of how academic programs could be structured to reflect real world 
application (Cox & Osguthorpe, 2003; Denner & Spector, 2007; Schwier, Campbell, & Kenny, 
2004; Tracey & Hutchinson, 2013). 

Finally, results of this study suggest that ISSs in military training organizations learn 
more about ID practices on the job rather than in their academic programs in education. I 
recommend future research that specifically asks the following questions: 1) What are the 
differences between how IDers perform in academic, industry, and military/government 
organizations; 2) What are the academic, industry, and military/government organizational 
environment factors that either enable or hinder Instructional Design duties; and 3) How could 
those identified factors be used to structure academic programs that support all three work 
environments? 

Conclusion 

As an introduction to ID practices and perceptions within military training environments, 
this study addressed the following questions: 1) What do Instructional Systems Specialists (ISSs) 
do (or what don’t ISSs do)?; and 2) What educational backgrounds do ISSs have that supports 
what they do as they work in the government training sector? Results of this study suggest that 


ISSs are not performing tasks associated with the Army regulatory design process of record or 
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the ADDIE process. In addition, more than half of ISSs participating in this study have 
educations oriented toward pedagogy or child/adolescent education rather than Adult Education. 

Based on these results, the Army may want to consider examining the job responsibilities 
intended for the ISS in comparison to other positions within the same career program. There is a 
call for a review of personnel management opportunities that leverage ID unique skills sets when 
making training development decisions. Understanding distinct job characteristics allows for the 
organization to leverage a professional skill set so that both product and process work toward the 
greater good of the organization. For military training organizations to maximize the value and 
organizational success of persons filling design positions, it is important for organizations to 
understand the abilities and capabilities of ISSs. Individuals with expertise and experience in 
instructional design have the skills to identify inefficiencies and resolve them with innovative 
solutions. ISSs are intended to be analytical experts that determine the best solutions to training 
and non-training problems. 

Based on the results of this study, it appears that ISS personnel are being used 
interchangeably with other training related personnel that do not (and are not required to) possess 
the requisite educational backgrounds and experiences in ID. Without the USAACE leadership 
leveraging ID specific skill sets and abilities, the Army may not have personnel with appropriate 
skill sets in ID-related positions. As the Army’s focus for training transitions from 
Counterinsurgency Operations (COIN) to Large Scale Combat Operations (LSCO) and Multi- 
Domain Operations (MDO), as well as from classroom learning to virtual learning environments, 
it is important for Army training organizations to become informed about ID and the distinct role 


it plays in effective training development. 
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Military organizational leaders may want to support a change in hiring practices when it 
comes to Instructional Systems Specialist (ISS) positions. Military and training development 
organizational leaders could increase the awareness for the necessity of hiring ISS personnel that 
have ID or Adult Education specific backgrounds and experience. Army training organizations 
may want to consider investing in a robust ID mentorship program for novice ISSs by those that 
have expert level experience in Instructional Design. Military training development leaders may 
also want to encourage and support Career Program 32 (CP32) professional development 
opportunities that emphasize ID or Adult Learning education and experiences. 

It is important for military and training development leaders to recognize the intricacies 
inherent in ID work so that when they direct training development decisions they are better able 
to determine which personnel is to perform which tasks and to what degree. Understanding the 
distinct differences between ID specific skills sets in a variety of training positions could have a 
positive impact on the quality and effectiveness of future Army training and mission 


performance. 
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Appendix A: Interview Protocol 


INTERVIEW PROTOCOL FOR PERCEPTIONS AND PRACTICES OF ARMY 


TRAINING INSTRUCTIONAL DESIGNERS 


Interview # 


Date i / 


Time 


Interviewee Name 
Army Location of interviewee 


Start with the Indiana University Informed Consent Script first 


After receiving the agreement to participate the study, you can start to ask the interview 
questions. 


PART ONE: 


RQI1: What type of education and/or training have instructional systems specialists (ISS) 
practitioners received as Instructional Designers? 


1. 


Would you describe your education background, please? 
(Look for School, Program, Major, Minor, level of degree) 
- Could you describe the types of courses you took in your degree program? 


Which Staff and Faculty courses have you completed? 
(Ask when each was completed?) 


Would you describe your history of continued education/professional development to 
date? 
(Look for CP-32 events, college classes, professional organizational participation, self- 
reading, conference attendance, etc.) 
- Would you describe the one of the projects that you have worked on within the 
last year? 


How many years have you worked in instructional design or a related field? 
How many years have you worked as an instructional systems specialist within an Army? 
- Could you describe in detail a project you’ve work on as an ISS in your current 
position? 


- Could you describe in detail a project you’ve work on during your career as an 
ISS? 
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RQ2: What IBSTPI Instructional Design tasks are performed most regularly by 
instructional systems specialists (ISSs)? 


RQ3: What is the perceived level of expertise requisite for performance of instructional 
design tasks by instructional systems specialists (ISSs)? 


6. Would you describe what you do on an average day of work? Use as much detail as 
possible. 
(Consider statements in comparison to Interviewer Checklist - IBSPTI Competencies) 
- Ask for examples of specific tasks/competencies mentioned. 
- Ask for an explanation of frequency tasks/competencies mentioned are performed. 
- Ask for explanation of expertise required to complete tasks/competencies 
mentioned. 
7. What are the most important skills you use in your position as ISS? What 
makes them the most important skills? 
8. In your opinion, what is the intended role of an ISS? 
9. Before we conclude this interview, is there anything else you would like to 
share? 
PART TWO: 


Unscripted questions 


** Tf participant wishes to discontinue study, ask if they would be willing to share why: 


Thank the participant for his/her participation 
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Appendix B: Email Invitation 
EMAIL TO POTENTIAL INTERVIEWEES 
Dear XXX, 


My name is Christina Parker, an Ed.D. candidate in the Instructional Systems Technology 
program at Indiana University under the direction of Dr. Thomas Brush. I am writing to invite 
you to participate in my dissertation research project on the perceptions and practices of 
instructional designers within Army training organizations. Your participation is requested 
because you have worked as an instructional systems specialists within the United States Army 
Aviation Center of Excellence (USAACE), Fort Rucker, Alabama. 


The research will be conducted using the interview process. The purpose of this interview is to 
gain a more in-depth understanding of survey responses. 


Depending on your schedule, the interview will be conducted sometime in the April-May, 2019 
timeframe. It will last approximately 40 minutes, be conducted in person or via an electronic 
communication tool such as skype or FaceTime or phone call and can be scheduled at your 
convenience. I will audio record the interview digitally, using the Rev application software. The 
audio records and transcripts will be kept safely with me. 


By participating in this interview, you will be asked several questions that seek to gain greater 
insight and understanding for your survey responses. Findings from this study may serve to 
improve human capital strategy development with the Army as well as provide insight into 
academic program designs. 


I sincerely hope that you will consider participating in my research study. If you are 
willing to participate in the interview, please contact me. If you have any questions about this 
study, please feel free to contact me as well. 


Sincerely, 


Christina Parker 

Ed. D. Candidate 

Instructional Systems Technology 
Indiana University 
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Appendix C: Informed Consent Script 


INDIANA UNIVERSITY INTERVIEW INFORMED CONSENT SCRIPT 
Instructional Design Perceptions and Practice in United States Army Training 
Organizations 


Principle Investigator: Christina Parker 
Instructional Systems Technology 
Indiana University 


Faculty Supervisor: Dr. Tom Brush 
Instructional Systems Technology 
Indiana University 


Script 


Welcome, and thank you for your participation in this study. My name is Christina, and I am an 
Ed.D. candidate at Indiana University in the Instructional Systems Technology program. This is 
my dissertation study. It aims to determine the perceptions and practice of instructional design 
task performance in United States Army Training Organizations. The study is conducted using 
the interview process. The purpose of this interview is to gain a more in-depth understanding of 
survey responses. This interview will last about 40 minutes. 


I would like your permission to audio record this interview, so I may accurately document the 
information you provide. If at any time during the interview you wish to discontinue the use of 
the recorder or the interview itself, please feel free to let me know. 


This study has no known risks. Please know that I will do everything I can to protect your 
privacy. Your identity or personal information will not be disclosed in any publication that may 
result from the study. The audio file and notes that are taken during the interview will be stored 
in a secure location. Your responses will remain confidential and will be used to develop a better 
understanding for the instructional designer task performance. 


Your participation in this interview is completely voluntary. If at any time you need to stop or 
take a break please let me know. You may also withdraw your participation at any time without 
consequence. Findings from this study may serve to improve human capital strategy 
development with the Army as well as provide insight into academic program designs. 


Do you have any questions or concerns before we begin? 
Then with your permission we will begin the interview. 
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Appendix D: Interviewer Worksheet 


INTERVIEWER WORKSHEET FOR PERCEPTIONS AND PRACTICES OF ARMY 
TRAINING INSTRUCTIONAL DESIGNERS 


Full = Performance discussed 
Partial = Performance discussed 


Full Partial Comments 
PROFESSIONAL FOUNDATIONS 
1. Communicate effectively in visual, 10 1 
oral, and written form. 
2. Apply research and theory to the 1 as 
discipline of instructional design. 
3. Update and improve knowledge, 10 0 Professional Development Discussed 
skills, and attitudes pertaining to the 
instructional design process and 
related fields. 
4. Apply data collection and analysis 1 1 
skills in instructional design projects. 
5. Identify and respond to ethical, 0 0 Competency not discussed within 
legal, and political implications of interviews by interviewees. Not 
design in the workplace. mentioned by interviewer. 
PLANNING AND ANALYSIS 
6. Conduct a needs assessment in 1 1 Generally addressed as not conducted 
order to recommend appropriate as projects are assigned by command 
design solutions and strategies. direction or command guidance 
7. Identify and describe target 0 2 Discussed as a “should be consider” 
population and environmental for design decisions but not normally 
characteristics. performed due to lack of time. 
8. Select and use analysis techniques 0 0 Competency not discussed within 
for determining instructional content. interviews by interviewees. Not 
mentioned by interviewer. 
9. Analyze the characteristics of 0 2 Discussed as a “should be consider” 
existing and emerging technologies for design decisions but not normally 
and their potential use. performed due to lack of time. 
DESIGN AND DEVELOPMENT 
10. Use an instructional design and 0 0 ADDIE is the TRADOC prescribed 
development process appropriate for a design process. Rapid Prototyping 
given project. seems to be the design process used 
in practice, however. 
11. Organize instructional programs 4 3 Interpreted as the use of 
and/or products to be designed, Storyboarding. 
developed, and evaluated. 
12. Design instructional interventions. 2 0 
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13. Plan non-instructional 0 0 

interventions. 

14. Select or modify existing Zz 0 Primarily within Staff and Faculty 

instructional materials. section with the modification of 
instructor courses/materials. 

15. Develop instructional materials. 0 0 

16. Design learning assessment. 0 0 

EVALUATION and 
IMPLEMENTATION 

17. Evaluate instructional and non- 4 12 Interpreted as “the review of the work 

instructional interventions. of others (1712s)” or the mention of 
validating designed materials. 
Validating is not conducted as 
“textbook” or was either discussed as 
how it should be executed. 

18. Revise instructional and non- 0 0 

instructional solutions based on data. 

19. Implement, disseminate, and 0 0 

diffuse instructional and non- 

instructional interventions. 

MANAGEMENT 

20. Apply business skills to managing 2 1 

the instructional design function. 

21. Manage partnerships and 9 0 

collaborative relationships. 

22. Plan and manage instructional 0 1 


design projects. 


Emerging Competency Trends: 


1. Data Entry within selected training development repository — 8 


2. Mentoring others in ID — 7 


3. Job/Task Analysis / Critical Tasks Site Selection Boards (CTSSBS) — 4 


4. Classroom Management/Teaching — 3 
5. Observing behaviors/environment — 3 


Randomly Mentioned Competency Trends: 


1. Student Management within LMS of record (Blackboard) 


2. Test Control 
3. Questioning Skills 


